Development of mobile applications: Synchronization with the server. Development of the back-end of mobile applications Creation of the back-end of the application

Offline in the past, being online today is a must. At least for the modern business world. Presentations of products and services of brands, online ordering and delivery, maintenance of a customer base, communication with customers, and much more - all this is simply impossible without an Internet presence. If you need an application, you must have both a Front-end (the web interface) and a Back-End (the server side of your application). And if you want to be able to edit the content of your application without the participation of developers, you need a good admin panel.

While Front-end in the sphere mobile applications created using technologies such as X-Code and Java, Back-end, where the database and all application logic will be stored, requires professional knowledge of the server-side programming language. good example is PHP, which is perhaps the most popular programming language that is used to develop almost any back-end. This is the undisputed leader.

There are many applications for PHP: static and dynamic websites + custom content management systems, social networks, specialized CRM systems, e-commerce software, and much more. Of course, there are free or cheap server parts and control panels. However, in many cases they do not provide the necessary level of convenience, customization and upgradeability.

Our programmers work with technologies that allow us to implement a wide range of solutions for various business goals, needs and requirements. We analyze system requirements for each project individually and we use various specialized server software for optimal performance of your mobile application.

If you're looking for a team that can lead you to the smartest and most cost-effective solution for building an app from scratch or restoring an existing one for a perfect user experience, look no further. Appsmob is ready to help you find the best solution For you.

Development of the server side of a client-server application begins with architecture design. A lot depends on the architecture: from the extensibility of the application to its performance and ease of support/maintenance.

First of all, you should determine how the data will be placed on the server and how requests coming from the client will be processed. It is also necessary to think over the organization of server-side data caching.

It is necessary to decide on data exchange protocols and data transfer formats.

API (application programming interface) - application programming interface. In a more understandable language, this is a set of requests to the server, which the latter understands and can give the correct answer. The API defines the functionality of the server logic, while the API allows you to abstract away how this functionality is implemented. In other words, the API is a necessary part of the overall client-server infrastructure.

Compare JSON and XML. Give an example of protocols depending on the type of application.

Multithreading

One of the key aspects in modern programming is multithreading. With the help of multithreading, we can separate several threads in the application that will perform various tasks at the same time.

Multithreading is a property of a platform (for example, an operating system, a virtual machine, etc.) or an application that a process spawned in the operating system can consist of several threads executing "in parallel", that is, without a prescribed order during time.

The essence of multithreading is quasi-multitasking at the level of one executable process, that is, all threads are executed in the address space of the process. In addition, all threads of a process have not only a common address space, but also common file descriptors. A running process has at least one (master) thread.

Multithreading (as a programming doctrine) should not be confused with either multitasking or multiprocessing, even though operating systems that implement multitasking tend to implement multithreading as well.

The advantages of multithreading in programming include the following:

Simplifying the program in some cases by using a common address space.

Less time spent on creating a thread relative to the process.

Improving process performance by parallelizing processor calculations and I/O operations.

Flow(thread) is a managed unit of executable code. In a thread-based multitasking environment, all running processes necessarily have a main thread, but there can be more. This means that a single program can run multiple tasks asynchronously. For example, editing text in a text editor while printing, because these two tasks are performed on different threads.

On a conventional processor, thread management is handled by the operating system. The thread is executed until a hardware interrupt occurs, system call or until the time allotted for it by the operating system expires. After that, the processor switches to the operating system code, which saves the state of the thread (its context) or switches to the state of another thread, which is also allocated time for execution. With such multithreading, it is enough a large number of CPU cycles are spent on operating system code that switches contexts. If thread support is implemented in hardware, then the processor itself will be able to switch between threads, and in the ideal case, execute several threads simultaneously for each clock cycle.

– Temporary multithreading (single thread)

– Simultaneous multithreading (multiple threads at the same time)

Multithreading, as a widespread programming and code execution model, allows multiple threads to run within a single process. These threads of execution share the resources of a process, but can also run on their own. The multithreaded programming model provides developers with a convenient abstraction of parallel execution. However, perhaps the most interesting application of the technology is when it is applied to a single process, which allows its parallel execution on a multiprocessor system.

This advantage of a multi-threaded program allows it to run faster on computer systems, which have multiple processors, a processor with multiple cores, or on a cluster of machines - due to the fact that program execution threads naturally lend themselves to truly parallel execution of processes. In this case, the programmer needs to be very careful to avoid race conditions and other non-intuitive behavior. In order to properly manipulate data, threads of execution must frequently go through a rendezvous procedure to process the data in the correct order. Threads of execution may also need mutexes (which are often implemented using semaphores) to prevent shared data from being modified at the same time or read during the modification process. Careless use of such primitives can lead to a deadlock.

Another use of multithreading, even for uniprocessor systems, is the ability for an application to respond to input. In single-threaded programs, if the main thread of execution is blocked by the execution of a long-running task, the entire application may be in a frozen state. By moving such long-running tasks to a worker thread that runs in parallel with the main thread, it becomes possible for applications to continue to respond to user input while the tasks are running in background. On the other hand, in most cases multithreading is not the only way to keep a program responsive. The same can be achieved through asynchronous I/O or signals in UNIX.

There are two types of multitasking: process-based And stream-based. The differences between process-based and thread-based multitasking are as follows: process-based multitasking is organized for the parallel execution of programs, and thread-based multitasking is for the parallel execution of individual parts of one program.

There are two types of streams:

Foreground threads or foreground. By default, every thread created through the Thread.Start() method automatically becomes a foreground thread. This type of thread provides protection for the current application from terminating. The common language runtime will not stop the application until all foreground threads have completed.

Background threads. This type threads, also known as daemon threads, are treated by the common language runtime as extensible execution paths that can be ignored at any time. Thus, if all foreground threads are terminated, then all background threads are automatically killed when the application domain is unloaded. To create background threads, you must set the IsBackground property to true.

Tell about the states of threads: running, suspended, running, but waiting for something.

Thread synchronization problem and shared resources.

Thread interaction

In a multithreaded environment, there are often problems associated with the use of the same data or devices by parallel executing threads. For solutions similar problems thread interaction methods such as mutexes, semaphores, critical sections, and events are used

Mutex is a synchronization object that is set to a special signaled state when not occupied by any thread. Only one thread owns this object at any time, hence the name of such objects (from English mutually exclusive access - mutually exclusive access) - simultaneous access to a shared resource is excluded. After all necessary actions, the mutex is released, giving other threads access to the shared resource. An object can support recursive capture a second time by the same thread, incrementing the counter without blocking the thread, and then requiring multiple releases. Such, for example, is the critical section in Win32. However, there are some implementations that do not support this and cause the thread to deadlock when attempting a recursive capture. This is FAST_MUTEX in the Windows kernel.

semaphores represent the available resources that can be acquired by multiple threads at the same time until the resource pool is empty. Then additional threads must wait until the required amount of resources is available again. Semaphores are very efficient because they allow concurrent access to resources.

Developments. An object that stores 1 bit of information “signaled or not”, on which the operations “signal”, “reset to an unsignaled state” and “wait” are defined. Waiting on a signaled event is the absence of an operation with an immediate continuation of the execution of the thread. Waiting on an unsignaled event causes the execution of a thread to be suspended until another thread (or the second phase of an interrupt handler in the OS kernel) signals the event. It is possible to wait for several events in the "any" or "all" modes. It is also possible to create an event that is automatically reset to an unsignaled state after waking up the first - and only - waiting thread (such an object is used as the basis for implementing the "critical section" object). Actively used in MS Windows, both in user mode and in kernel mode. There is a similar object in the Linux kernel called kwait_queue.

Critical sections provide synchronization like mutexes, except that the objects representing critical sections are accessible within the same process. Events, mutexes, and semaphores can also be used in a single-process application, however, implementations of critical sections in some operating systems (for example, Windows NT) provide a faster and more efficient mechanism for mutually exclusive synchronization - the acquire and release operations on the critical section are optimized for case of a single thread (no contention) in order to avoid any system calls leading to the OS kernel. Like mutexes, an object representing a critical section can only be used by one thread at a time, making them extremely useful in restricting access to shared resources.

Condition variables(condvars). Similar to events, but they are not objects that occupy memory - only the address of the variable is used, the concept of "contents of the variable" does not exist, the address of an arbitrary object can be used as a condition variable. Unlike events, setting a condition variable to the signaled state has no consequences if there are currently no threads waiting on the variable. Setting an event in a similar case entails storing the "signaled" state within the event itself, after which subsequent threads that wish to wait for the event continue execution immediately without stopping. To make full use of such an object, the operation “release the mutex and wait for the condition variable atomically” is also necessary. Actively used in UNIX-like operating systems. Discussions about the advantages and disadvantages of events and condition variables are a prominent part of the discussions about the advantages and disadvantages of Windows and UNIX.

I/O Completion Port(IO completion port, IOCP). Implemented in the OS kernel and accessible through system calls, the "queue" object with the operations "put the structure to the tail of the queue" and "take the next structure from the head of the queue" - the last call suspends the execution of the thread if the queue is empty, and until no other thread will make the put call. The most important feature of IOCP is that structures can be placed in it not only by an explicit system call from user mode, but also implicitly inside the OS kernel as a result of the completion of an asynchronous I / O operation on one of the file descriptors. To achieve this effect, you must use the "associate a file descriptor with IOCP" system call. In this case, the structure placed in the queue contains the error code of the I / O operation, and also, in the case of success of this operation, the number of actually entered or output bytes. The implementation of the completion port also limits the number of threads executing on a single processor/core after a structure is received from the queue. The object is specific to MS Windows, and allows the processing of incoming connection requests and data chunks in server software in an architecture where the number of threads can be less than the number of clients (there is no requirement to create a separate thread with resource costs for each new client).

Thread pool

Tell about the thread pool

Our company offers services for creating the server part of mobile business applications and client web services operating in high-load environments. When developing each project, we try to apply an individual approach so that the resulting product becomes the most optimal solution specific customer goals.

If you are using a complex application that stores and / or processes data on a server, then it is backed by a Back-end - a software package hosted on a web server and working with an application, which in this case is called Front-end. An application hosted on a server can work simultaneously with a large number of clients, which imposes requirements on high speed and security of its operation.

Often the server side of an application is written in PHP language, as the most popular for such solutions. To implement simple server tasks can be used standard systems, but for more specific ones, it is already required to develop your own solution or add-ons over standard systems.

Principles of developing a server for a mobile application

Our programmers work with technologies that make it possible to implement wide range solutions for any, even very high load and various directions. We also create separate server solutions for individual tasks.

Organizational control

Each project is created by a separate group of specialists responsible for all stages of development and delivery of the project on time.

Programming

Designing the server architecture is the most important step during which databases are created and the necessary algorithms are formed.

Testing

The software part should work without errors and failures. This is the responsibility of the testers who carry out the verification of the system.

Technical support

Our employees perform full-fledged technical support of programs, which allows you to quickly eliminate deficiencies and make updates.

Development features

In order to competently develop the server part of the application, certain skills and knowledge of the programming language used on the server are required. As practice shows, the client server applications created in PHP. He is the undisputed leader in this field. More than half of the sites in the world are written in PHP, it is convenient for development and support.

Framework

This software platform allows you to make the project more scalable and flexible. Nevertheless, the framework should be chosen as correctly as possible, therefore, a deep analysis of the working documentation of the project is required, which will subsequently help to develop a high-quality product.

There are other languages ​​used for back-end development. For example, server applications created in Delphi environment. Due to it, the program has improved debugging. Moreover, it is easier to develop unique programs, it provides visual creation. All this allows you to make a clear and convenient interface.

Java server applications are no less popular. They are easily complemented, easily executed on different platforms, and have a high level of security.

Another commonly used language. With it, server applications are created easily, quickly and at no extra cost.

Almost all modern companies have their own virtual offices. The website can be either calling card, or a portal or online catalog with the possibility of placing orders.

Business processes in this case are dependent on web servers, namely their ability to withstand attacks, hacking attempts and external negative impacts, as well as performance sufficient for many simultaneous requests.

Stages of web service development

Creating applications for different market segments, we organize our work according to a single principle - we break the whole process into individual steps, the progress and results of which are reported to customers. Thus, the server for the mobile application is developed in the same way.

1. Idea development

Up to 2 weeks

At this stage, the foundation is being created, giving an idea of ​​what will be laid down and in what direction it will develop.

2. Project evaluation

2-3 weeks

Our experts evaluate the time and cost of the work, then a preliminary proposal for development is drawn up.

3. Terms of reference and contract

Up to 2 weeks

After discussing with the customer all the nuances of the process and drawing up a detailed TOR, a contract is prepared and signed.

4. Interface development

2-3 weeks

Designers are responsible for creating the interfaces needed when writing program modules.

6. Testing

2-3 weeks

Comprehensive verification of received software solution produced by testers through a set of appropriate tools.

7. Completion of the project

Up to 2 weeks

Within the agreed timeframe, a ready-made, thoroughly tested web service is handed over to the customer.

our team

Through the analysis of commercial activities and the needs of our customers, we create real-world products that help solve a range of business problems. Usage modern technologies provides a wide range of possibilities for the implementation of server software, which guarantees the high performance of the corresponding mobile applications. Our team is represented by:

Project Managers

These employees interact with both clients and developers, providing communication between them. They monitor the implementation of both the already planned actions and the necessary improvements.

Designers

In their work, our specialists take into account the requirements for building interfaces for operating systems iOS and Android, so the released applications work correctly on different devices.

Developers

In order to optimize the performance of mobile applications, programmers analyze their system requirements and create specialized server software.

Testers

Thorough testing is a guarantee of the quality of the finished product and a guarantee of the security of stored and processed data. These specialists use different tools and an effective methodology.

What services do we create

Being a software built into the site or an independent program, the web service is used to perform tasks related to advertising, analytics, business planning and promotion. In this regard, it is necessary to decide what type of resource will be the best solution.

Information projects

Designed to accommodate diverse content.

Thematic sites

Almost all of their pages are devoted to one topic. The demand for them is still quite high.

news sites

They inform about various news within the framework of one or more topics, reflecting the main areas of life.

Blogs

The level of popularity of these resources is constantly growing. Like news sites, they convey this or that information to the Internet community, but in this case, the authors express their personal opinion.

Social projects

These include specialized social networks, communities, forums, etc.

Forums

Created to discuss various news, products / services, etc. They can be both narrowly focused and diverse.

Social media

These resources have a multi-million audience. Their main task is to provide Internet users with the opportunity to communicate online via text / voice messages and video communications.

Various web services

Received today wide use, they are divided into several types.

Catalogs

Postal services

Provide users with all the features and benefits Email, including viewing, sending, editing letters and documents, etc.

Search engines

They are used to search for sites and various information on specific requests.

Notice boards

These are web resources where network users place their advertisements for the sale and purchase of services within various topics.

Hosting sites

Designed for temporary storage of files. Some of them provide an opportunity to get acquainted with the data before downloading.

FAQ

Below we offer answers to questions that are often asked to our specialists. If you do not find the information you are looking for here, please post your question here. form and we will definitely answer it.

How long can it take to create an application and a web server?

On average, this work lasts from 9 to 20 weeks. It all depends on the complexity of the task being implemented.

BACKUPS

Why backups are needed on a mobile platform

Experts know how unreliable 1C mobile applications are sometimes: errors can occur at any time, due to which user bases simply collapse. At the same time, we are faced with the unreliability of the devices themselves: they can be broken, lost, stolen, and users want to keep their data. And up to version 8.3.9 we didn't have a platform backup mechanism.

Since users didn't have a "save a copy" button before, the developers of the Boss app had to make backups themselves. How did we do it?

We save the database data in the form of XML.

It is advisable to offer the user several options for storing copies - first of all, it is convenient for customers, they can choose the best option for themselves: upload to the cloud, send it to their mail, save it on the device.

Thus, the developers additionally insure themselves. If something went wrong, and the mechanism for creating copies on Google Drive or Yandex Drive suddenly broke down, you can always tell the user that the developer is currently dealing with the error, but for now he can save the data in an alternative way. And users are satisfied because they can be calm about their data.

Necessarily focus on cloud services, because if the device is lost or broken, and the user saved a copy on the same device, then the data will be lost.

Also we be sure to remind the user of the need to create backups.

How to save copies if the configuration changes?

When we talk about a mass solution, about an application that is constantly changing, developing and improving, we must take into account the behavior of customers. The user may want to restore a backup saved in old version application, where there were no details. And then the task arises: to read the data, then fill in the data according to the update logic from the old version of the application. How to do it? In addition to the data, save the data structure itself, so that later you know how to read them.

There are several options for storing this data structure, including storing it in the configuration itself. That is, with the release of each new version, save the metadata structure of the previous version in the layout in the configuration.

Do not forget that in a mobile application, the configuration should not grow just like that, we must value the place in it, we must make it as compact as possible. But the application is developing, and there will be many such layouts, and over time there will be more and more of them.

Therefore, in the case of a mobile application, another way is preferable - save the metadata structure directly in the data file. At the output, we get such a file, where at first we store some auxiliary data - the configuration version, configuration scheme, sequence boundaries, and then we write the user data themselves in XML format. Moreover, in the "Auxiliary data" section of the file, you can also store other important data that for some reason could not be written to XML.

We take the data schema that was saved to the file, and based on it we build the XDTO package for reading the file. We create a similar object in the database, fill it in, perform completion processing when updating, and save the finished object to the database.

Below in the picture you can see a hint on how to beautifully write the XDTO model of these configurations. The company that released the Boss application experimented with this, found several ways, but settled on just this option for writing the metadata schema. When the data file itself is opened, you can see the usual structured XML, readable, which lists all the metadata of the application.

// Write configuration schema XDTO Model = XDTO Factory.XDTO Model Export("http://v8.1c.ru/8.1/data/enterprise/current-config"); FactoryXDTO.WriteXML(FileUpload, ModelXDTO); // Read configuration schema ModelXDTO = FactoryXDTO.ReadXML(ReadingXML, FactoryXDTO.Type("http://v8.1c.ru/8.1/xdto","Model")); Unload Factory = New XDTO Factory(XDTO Model);

To protect the user, it is necessary to ask him again if he needs to restore the backup. Maybe he was just experimenting and clicking buttons on everything in the application :) And now his current data may be lost. Therefore, when performing potentially "dangerous" actions, we always clarify whether he really wants this, and how this should happen. The user must be aware of his actions.

There must be a mechanism for creating backups when we are talking about an offline solution, when the user has all the data stored exclusively on a mobile device: the user can lose his device, and then the data will be lost. And it would seem that if the application does not work offline, but is connected to a central server, then the user should not have such a problem, because if the device is lost, he will connect to the server, receive all his data from the server again, and everything will be ok.

However, users do not always use backups in the way we expect from them :) They very often use them in order to simply “roll back” data back. This is really a very strange behavior, but mobile application users are too lazy to figure out where they could make a mistake when entering data, and they simply roll back the data and re-start the data for the current day. After analyzing the statistics of working with the Boss application, we realized that this is a normal practice and such user behavior is more common than we could have imagined.

And if you use synchronization with other devices, then you must process this. There are several solutions here:

• break the connection with the server, specifying that the data on it will remain as it was, and the copy will be restored only on the user's device;
• it is better for the user to let him restore a copy on all devices at once, having previously prescribed such mechanisms.

There is one more point here. Until now, we saved backups ourselves, controlled the entire process, caught the user’s actions right in the code when he clicked the “save a copy” button. All this can be processed later. In platform 8.3.9, it became possible to save backups using the platform's tools. And the user does it without our knowledge. If synchronization with a central database is used, then such a scenario must be processed. We must somehow find out on our server that the user has restored a previously saved copy and must give him some kind of decision. We can't afford to let the data get out of sync.

EXCHANGE

When we talk about a private solution on a mobile platform, we usually have a customer who, for example, wants to use a mobile platform for his sales agents and have them exchange data with a central database. Everything is simple here: one database, several devices, you raise the server, set up communication with it. So the problem of exchange between devices is solved easily.

But if we are talking about a mass application, where there are many databases, each of which has a lot of users, the situation becomes more complicated. Users have downloaded the app from the market and they want to sync with each other. For example, a husband has downloaded a personal finance app and now wants his wife to connect too so they can work together on the same app. There are many users, the application is developing, growing, and there is a need for a large, very large number of databases. How to organize all this? Users will not personally contact developers to create a separate database for them and enable synchronization. They want to push a button and make everything work right away. At the same moment.

How to proceed? This is where the data separation mechanism comes to the rescue. It allows you to organize a single database, where there is one common configuration, but at the same time, an unlimited number of user databases are stored within one common database.

The best part is that you can add users dynamically, programmatically, without our participation. In reality, users simply click on the “register on the server” button, and everything happens by itself: a personal database is created for him on the server, and he can immediately start working in it.

How to do it? The first and simplest solution is to write your own server database with this mechanism. When our company started to make the Boss application and exchanges in it, we did just that in the first version: we wrote a server database with a data sharing mechanism. Everything worked, especially since there was nothing complicated - the base separator is a common attribute.

But then we realized that we were reinventing the wheel :) In fact, there are already turnkey solution, and it already takes into account moments that we have not even thought about. This is 1C: Fresh.

The scalability of the service is thought out here: what to do when there is a lot of data and databases, how to grow with it all. There's a moment here about creating backups data areas: that is, we do not just backup one common database, we make copies of a specific user. Moreover, the mechanism there is such that copies are made only when they are really needed. If the user has not logged into the database for a week, then we do not make copies for him, because nothing has changed there. Another feature of Fresh is that the service has a mechanism to reduce the load on the server, and this is very important when you have a lot of databases.

In general, Fresh is something new and interesting for us. Slowly we are trying to figure it out, but for the most part we are just happy with his work.

Data transfer. How to implement it for sharing between devices

The platform provides two mechanisms - SOAP and http services. There are nuances here on how to access these services when the data sharing mechanism is enabled. In particular, you need to add parameters that indicate the specific number of the realm you are accessing, because the platform cannot determine which database to access from the username. In addition, the same user can work with several databases within a single database (see picture).

As for services, the Boss application implements instant exchange: One user enters data and another receives it. Mobile application users are used to the fact that everything happens instantly, so we thought about how better service to use - SOAP or http. Connection speed played a key role. In http, the connection speed is much higher, and when connecting via SOAP, we get a service description that is heavy and takes a long time to load. The platform has a way to store a description of the service, but because of the parameters we add dynamically, we can't use WS links. In addition, accessing http services is more convenient and flexible, in our experience.

So, our goal is to implement real-time exchange. That is, we try not to make it so that the user has to go somewhere, click on some button, think about how up-to-date his data is, whether he should update it ... Users should always have data up-to-date. They are so used to working in messengers - one sent the data, the other immediately received it. Everything happens instantly. The same applies to applications related to business: one seller has issued a sale, the other should immediately see the current situation, without taking any action.

Therefore, the Boss application uses background jobs for exchanges. After each data entry into the database, a background task is launched that initiates the exchange. The first part is to send the data to the server. The other devices then need to know that there is new data. For this we use PUSH notifications. This scheme is already working and it works fast enough.

But we wanted even faster, because we work in real time and we usually have little data. We have small XML, but at the same time we send a message with this data from the first device to the server, the server sends PUSH to another device, and then the second device, after receiving PUSH, initiates the exchange on its part, accesses the server and requests data, receives this data and then sends a response that the data has been received. This is a long time, but there was very little data itself.

We thought about how this process can be accelerated.

To do this, we figured out what PUSH contains, how it can still be used. It turned out that PUSH contains fields such as data and text. The documentation for iOS and Android indicates the limits on the size of PUSH messages, but it seemed to us that this was not enough and we wanted to figure it out by ourselves. And we checked that for iOS the sum of allowed characters is 981 characters, and for Android it is 3832 characters. In the latter case, it is quite possible to use the restriction; one or several base objects can be pushed into such a volume. And then the company's developers changed the scheme a bit. When there is little data, we send it from one device, receive it on the server, pack it in PUSH and send it directly to another device. The scheme has become shorter, and the exchange has become even faster :)

An important point of using PUSH is that it does not annoy users.

It is very easy to get rid of this situation: just don't send a lot of PUSH messages to the user :) If he is currently working in the application, you can send a lot of messages. When the platform is running, the user does not see PUSH, everything happens automatically for him. But when the application is closed, the client has a lot of unread messages. Therefore, in no case should you send the next PUSH until a response is received from the device that the application is running, active and the previous PUSH has already been processed.

Another nuance of the exchange is work via the web. We need to use asynchrony as much as possible. You can't work as usual - write code - call a function - wait for it to execute - get a response - and everything is ok. If you work via the web, you will still encounter certain limitations, such as unstable Internet, timeouts when performing long operations. Therefore, it is necessary to think over the architecture in advance.

Let's look at the example of device registration, what happens in the application when the user wants to register. He keeps records for a while, he entered a lot of data, but then he wants the seller to also work with this database. The user clicks on the "register" button. At first, everything was very simple: they took his data, recorded it on the server, and, please, you can work and connect users. But then we ran into a situation where for some users the databases on the device had already grown greatly by the time of registration. And this scheme no longer worked, because. while the entire database was being recorded on the server, the connection timeout was triggered or the Internet was simply interrupted. Therefore, we have replaced one synchronous call with many short calls. Data is now shared rather than being transferred all at once. We do not wait in any case while the server will process and write data. We sent the data, received a response that the data was received, closed the connection. Periodically, it is necessary to poll the server, what is happening there and how, and in the meantime, a background task is running on the server, which writes the received data. This makes a lot of server calls, but we have a guarantee that everything will go well. And neither timeouts nor Internet instability will prevent you from uploading all the data to the server.

UPDATES

Sharing between devices with different app versions

Since we are talking about a mass application that is released to the markets, it is necessary to take into account some features of the update and data exchange process.

If you have released an application for one enterprise and decide to update it, then usually you just give a command that all employees unanimously install the new application. With users who downloaded the application from the market, this cannot be done. You can't tell them what to do at all. For example, they work in the application and do not want to update it now or ever. They do not have auto-update, so it is quite common for several devices to be connected to the central database, and all of them with different versions. Another reason for this phenomenon is the publication time in the markets: it is different for iOS and Android. We often implement key things like fixing critical bugs and don't want to wait two weeks for iOS to review new version, we want to at least only for Android, but release an update right now.

We have no right to command users. If they want, they update, and if not, they do nothing. The picture shows the ratio of installations of the Boss app by versions in GooglePlay, as well as statistics from our server - the real ratio of app versions that are installed on devices that have exchanged data with the server over the past week. Here is a set to work with. These are different versions and different metadata. And we need to organize a normal exchange at the same time :)

Developers face the following tasks:

• It all needs to work. Users shouldn't feel bad about forgetting to update. They shouldn't notice it at all. Updated - it became better, well, good.
• We must ensure the safety of the data. For example, one user has a directory and a new attribute, while another does not yet. Moreover, if a user who does not have new details changes something on his device, then data on other devices should not be lost.
• It is necessary to ensure that the data is updated when we switch to a new version. When the user decides that he is ready to upgrade, he should automatically have all the new information that he did not have just because he had the old version.

How did we do it?

1. We use 2 exchange plans on the server. The first is for sharing between devices, and the second is for updates. For example, we sent a directory to the user, but it does not have units of measure, that is, incomplete data. We must remember this. And when it is updated, we must send it all the information that it did not have. For this, a second exchange plan is needed.

2. To write and read objects, we use the same mechanism that is used for backups, that is, we save the metadata version. In this case, we are working with the server, and we can afford to add anything we want directly to the configuration, so we simply add metadata schemes to the configuration in the form of layouts as the application develops.

How to monitor massive errors in the exchange and on the server

First, you need to control the availability of the server itself. With servers it happens - they fall. We did not invent anything special to monitor, but simply found a bot in a telegram that screams if something is wrong. He checks the server's performance every minute, and if suddenly the server is unavailable, he starts screaming, the admins see this and raise the server.

We also collect an error log from the log. Also nothing supernatural - just every three hours we collect an error log, send them to the mail, periodically review them. This helps to see common problems and some exceptional situations. It is not difficult to view mail, track and quickly fix errors. But this allows you to quickly identify and solve problems that can grow with the growth of databases.

Yet important point- be sure to give the user the opportunity to "complain". This improves our status in their eyes and saves us. There are users, as we call them, "hysterics" who, at the slightest mistake, start sending us a bunch of messages by mail that nothing works, the database does not load, everything is terribly bad. But sometimes they really save us, because sometimes they find bugs that the rest have not yet miraculously discovered, serious bugs.

The user should not be scared. No scary messages, nothing else. They need to explain everything beautifully and offer to complain. And we promise to solve everything. Then the users are satisfied, because they see that they are taken care of, and immediately believe that they will be helped :)

This article was written following the results of the report read at the INFOSTART EVENT 2016 DEVELOPER conference. More articles can be read.

In 2020, we invite everyone to take part in 7 regional meetups, as well as the anniversary INFOSTART EVENT 2020 in Moscow.

A significant part of modern applications for mobile platforms(iOS, Android, etc.) works in tandem with the server. An application with stale data loses its usefulness. Therefore, it is important to ensure that the data from the server to the device is constantly updated. This applies to offline applications that should work without the Internet. For completely online applications, which do not work (or are useless) without the Internet (for example, Foursquare, Facebook) have their own specifics, which are beyond the scope of the current article.

Using one of our offline applications as an example, I will tell you what approaches we used to synchronize data. In the first versions we have developed simple algorithms and, in the future, with experience, we improved them. A similar sequence is presented in the article - from simple obvious practices to more complex ones.

It should be clarified that the article deals with data transfer only in one direction: from the server to the device. Here the server is the data source.

General provisions for all approaches

For example, we will consider transferring a directory of dishes (“dishes”) to the device. We will assume that the device makes a request to the url “/service/dishes/update”, the exchange takes place via the http protocol in JSON format ( www.json.org). The server has a “dishes” table with the following fields: id (record identifier), name (dish name), updated (dish update time, it’s better to immediately support timezone, “YYYY-MM-DDThh:mm:ssTZD”, for example, “1997 -07-16T19:20:30+01:00”), is_deleted (sign of a deleted entry).

Remark regarding the presence of the last field. By default, its value is 0. In an application where entities are synchronized between the client and the server, it is not recommended to physically delete data from the server (so that there are no errors). Therefore, is_deleted = 1 is set for deleted dishes. When an entity with is_deleted = 1 arrives at the device, it is deleted from the device.

With either approach, which will be discussed below, the server returns an array of objects to the JSON devices (may be empty):

[
(id: ,name: .updated: ,isDeleted: },…
]

Server response example:

[
(id: 5625,name: "Bread",updated: "2013-01-06 06:23:12",isDeleted: 0),
(id: 23,name: "Cooked semolina",updated: "2013-02-01 14:44:21",isDeleted: 0),(

name: "Fish Soup",

updated: "2013-08-02 07:05:19",

Principles of updating data on the device

1. If an element arrived that is on the device and isDeleted = 0, then it is updated
2. If an element has arrived that is not on the device, and isDeleted = 0, then it is added
3. If an element arrived that is on the device and isDeleted = 1, then it is deleted
4. If an element arrives that is not on the device and isDeleted = 1, then nothing is done

Approach 1: Always sync everything

This is the easiest method. The device requests a list of dishes from the server, and the server sends the entire list. The whole list comes up every time. Not sorted.

Request example: null, or "()"

Advantages:

• the logic on the server is simple - we always give everything
• the logic on the device is simple - we always overwrite everything

Disadvantages:

• if you request the list frequently (every 10 minutes), then there will be a lot of Internet traffic
• if you request the list rarely (once a day), then the relevance of the data will be violated

Application area:

• for low traffic applications
• transmission of very rarely changing data (list of cities, categories)
• transfer application settings
• at the beginning of the project for the very first mobile application prototype

Approach 2: Only updated

The device requests the list of dishes updated from the previous synchronization. The list comes sorted by “updated” in ascending order (optional, but handy). The device stores the “updated” value for the most recently sent dish and sends it to the server in the “lastUpdated” parameter on the next request. The server sends a list of dishes that are newer than "lastUpdated" (updated > lastUpdated). On the first request to the server, “lastUpdated” = null.

Request example: ( lastUpdated: “2013-01-01 00:00:00” )

In the diagram: “last_updated” is the value that is stored on the device. Usually, a separate table is created on the device to store these “last_updated” values ​​for each entity (dishes, cities, organizations, etc.)

This approach is suitable for synchronizing simple linear lists, which have the same arrival rules for the device for all devices. For a more selective sync, see Approach 5: Sync knowing what's already on the device.

Usually this approach covers most of the needs. Only new data comes to the device, you can synchronize at least every minute - the traffic will be small. However, there are problems associated with restrictions. mobile devices. These are memory and processor.

Approach 3: Sync in chunks

Mobile devices have few random access memory. If there are 3000 dishes in the directory, then parsing a large json string from the server into objects on the device may cause a memory shortage. In this case, the application will either crash or not save those 3000 dishes. But even if the device was able to digest such a string, then the performance of the application at the moments of synchronization in the background will be low (interface lags, not smooth scrolling, etc.). Therefore, it is necessary to request the list in smaller portions.

To do this, the device passes another parameter (“amount”), which determines the portion size. The list is sent necessarily sorted by the “updated” field in ascending order. The device, similarly to the previous approach, remembers the “updated” value of the last sent entity and passes it to the “lastUpdated” field. If the server sent exactly the same number of entities, then the device continues to synchronize and makes a request again, but with the updated “lastUpdated”. If the server sent fewer entities, it means that it has no more new data, and the synchronization ends.

In the diagram: “last_updated” and “amount” are the values ​​that are stored in mobile application. “last_item” – the last entity (dish) sent from the server. It is newer than this value that the next list will be requested.

Request example: ( lastUpdated: “2013-01-01 00:00:00”, amount: 100 )

Advantages:

• The device receives as much data as it can process at one time. Serving size determined by practice tests. Simple entities can be synchronized 1000 at a time. But it also happens that entities with a large number of fields and complex saving processing logic are synchronized normally no more than 5 pieces.

Disadvantages:

• If there are 250 dishes with the same updated, then with amount = 100 the last 150 will not be delivered to devices. This situation is quite real and is described in the following approach.

Approach 4: Correct chunk timing

In the previous approach, it is possible that if there are 250 dishes in the table with the same “updated” (for example, “2013-01-10 12:34:56”) and the serving size is 100, then only the first 100 records will come. The remaining 150 will be cut off by a hard condition (updated > lastUpdated). Why will this happen? When querying the first 100 records, lastUpdated will be set to “2013-01-10 12:34:56” and the next query will have the condition (updated > “2013-01-10 12:34:56”). Even softening the condition (updated >= “2013-01-10 12:34:56”) will not help, because the device will then endlessly request the first 100 records.

The situation with the same “updated” is not so rare. For example, when importing data from text file the "updated" field was set to NOW(). Importing a file with thousands of lines can take less than a second. It may also happen that the entire directory will have the same “updated”.

To fix this, you need to use some dish field that would be unique at least within one moment (“updated”). The “id” field is unique throughout the entire table, so you should additionally use it in synchronization.

In summary, the implementation of this approach looks like this. The server returns a list sorted by “updated” and “id”, and devices request data using “lastUpdated” and the new “lastId” parameter. For the server, the selection condition becomes more complicated: ((updated > lastUpdated) OR (updated = lastUpdated and id > lastId)).

In the diagram: “last_updated”, “last_id” and “amount” are the values ​​that are stored in the mobile application. “last_item” – the last entity (dish) sent from the server. It is newer than this value that the next list will be requested.

Approach 5: Synchronization with the knowledge of what is already on the device

The previous approaches do not take into account the fact that the server does not really know how successfully the data was stored on the device. The device could simply not save some of the data due to unexplained errors. Therefore, it would be nice to receive confirmation from the device that all (or not all) dishes have been preserved.

In addition, the user of the application can configure the application in such a way that he needs only a part of the data. For example, a user wants to synchronize dishes from only 2 cities out of 10. This cannot be achieved with the synchronizations described above.

The idea behind the approach is the following. The server stores (in a separate table “stored_item_list”) information about what dishes are on the device. It can be just a list of "id - updated" pairs. This table stores all lists of “id – updated” dish pairs for all devices.

The device sends information about the dishes available on the device (a list of “id – updated” pairs) to the server along with a synchronization request. When requested, the server checks which dishes should be on the device and which ones are currently available. After that, the difference is sent to the device.

How does the server determine which dishes should be on the device? In the simplest case, the server makes a request that will return a list of “id - updated” pairs of all dishes (for example, SELECT id, updated FROM dishes). In the diagram, this is done by the “WhatShouldBeOnDeviceMethod()” method. This is the disadvantage of the approach - the server has to calculate (sometimes making heavy sql queries) what should be on the device.

How does the server determine what dishes are on the device? It queries the "stored_item_list" table for this device and gets a list of "id - updated" pairs.

By analyzing these two lists, the server decides what should be sent to the device and what should be deleted. In the diagram, this is “delta_item_list”. Therefore, there are no “lastUpdated” and “lastId” in the request, their task is performed by “id - updated” pairs.

How does the server know about the dishes available on the device? Added to the request to the server new parameter“items”, which contains a list of id items that were sent to the device in the last synchronization (“device_last_stored_item_list”). Of course, you can send a list of id of all dishes that are on the device, and not complicate the algorithm. But if there are 3000 dishes on the device and they will be sent every time, then the traffic costs will be very high. In the vast majority of syncs, the “items” parameter will be empty.

The server must constantly update its “stored_item_list” with the data that came from the device in the “items” parameter.

You should implement a mechanism for clearing server data in stored_item_list. For example, after reinstalling an application on a device, the server will assume that the data is still up to date on the device. Therefore, when installing the application, the device must somehow inform the server so that it clears the stored_item_list for this device. In our application we send additional parameter"clearCache" = 1 in this case.

Conclusion

Summary table on the characteristics of these approaches:

№	An approach	Traffic volume(5 - large)	Labor intensity of development(5 - high)	Device memory usage(5 - high)	Correctness of data on the device(5 - high)	You can select a specific device
1	Everything is always synchronized	5	1	5	5	No
2	Synchronized only updated	1	2	5	3	No
3	Synchronization in chunks	1	3	1	3	No
4	Correct synchronization in chunks	1	3	1	3	No
5	Synchronization with the knowledge of what is already on the device	2	5	2	5	Yes

“Data correctness on the device” is the probability that the device has all the data that was sent by the server. In the case of approaches #1 and #5, there is 100% certainty that the device has all the data that is needed. Otherwise, there is no such guarantee. This does not mean that other approaches cannot be used. Just if the device has a part data will be lost, then it will not work to fix it from the server (and even more so to learn about it on the server side).

Perhaps, in the presence of unlimited tariffs for the Internet and free wifi problem restrictions on traffic generated by a mobile application will become less relevant. But for now, we have to go to all sorts of tricks, come up with more “smart” approaches that can reduce network costs and increase application performance. It doesn't always work. Sometimes it happens “the simpler the better”, depending on the situation. I hope this article can help you find an approach that will come in handy.

There are surprisingly few descriptions of server synchronization and mobile devices. Moreover, there are many applications that work according to this scheme. For those interested, a couple of links.