CTS full-process automation solution discussion
functional vision
cts full-process automation. It roughly includes the following modules:
- Version build
- Forced OTA upgrade
- Auto retry
- Report + log backup and analysis< /li>
- Mail sending
Program pain points
| Goal< /th> | Solution | Problem level & pain points |
|---|---|---|
| Version construction | Current plan: Python library paramiko remote ssh login Details: Win-side python script paramiko library remote ssh login to the Linux compilation server, write the one-click compilation command to a python file, and then execute it “One-click compile command” py file |
Problem level: low Sometimes the version cannot be downloaded and manual intervention is required. Automatic construction is not a problem, something that daily has already achieved. |
| Forced OTA upgrade | Current plan: The plan needs to be studied and discussed, and how to change the configuration More The set-top box upgrade has not been implemented. Previous plan: 1. Copy the OTA package of the compilation server to the Apache server path on the win side (prepare the upgrade rule file and configuration, the upgrade configuration is to avoid cyclic upgrades) 2 .adb install Modify the configuration apk to modify the versionserver, restart the forced upgrade (at the same time upgrade the configuration) After the configuration is separated, there is aidl, so the jni client is implemented, and the apk is completed in the form of a service accepting parameters, so that the configuration can be written by am satrtservice. But it is not available now. The specific reason needs to be discussed (Android9 prohibits background startup service) |
Problem level: high Cannot modify the configuration with the adb command. This module is about It is equivalent to abolishing The conjecture of the scheme: am satrt activity and then -es pass parameters? In addition, the OTA upgrade data is full. How to clear it under selinux:enable (involving cts to copy media files)? Guide key (you don’t need to burn after burning once?) What is the version difference between OTA upgrade and USB burning? |
| Automatic retry | Current plan: Shell script calls gnome terminal to implement pseudo-child process, the beginning of new retry From the cts-tf >lr command result monitoring Why use the current solution? : Because of the following attempts: 1.python subprocess: can start the cts-tradefed script, can start the test, cannot add the –shard-count parameter 2. Shell directly./cts-tradefed xxx Call the script to start the test, you can run the test, but you cannot exit. If you have run the cts test, you will find that this script is always waiting for input cts-tf> 3. Another important point is that the above two standard output cannot be redirected, so currently we use ./cts-tradefed lr The method of monitoring results, guess there are other methods, please discuss |
Problem level: high Generally, I set up retry 5 times, so I start by opening seven or eight terminal test machines There is a noticeable freeze. In addition, the retry scheme cannot be called remotely via ssh (running the script will not generate a new gnome terminal), the reason is unknown. I believe this is the lowest retry method. Please discuss There is an important issue if manual intervention is excluded (how to determine that this is the robust one we want to give to developers Analysis report), will be discussed in the next module. |
| Report + log backup and analysis | Current plan: 1. Copy the results of the Linux test machine to the backup server 2.requests and bs4 parse the report html, filter and categorize the failed items, and generate Markdown HTML (in order to synchronize ICenter and look better than the text?) |
Problem level: low Although the ideal of full automation is beautiful, the reality that we have to face is: In the early version, the number of failed items ranges from dozens to five to six hundred. How does the script determine what status the report is? If it can be sent out, this needs to be focused discussion. According to experience, what developers need to analyze should not be failures caused by the environment (for example, the network is disconnected, the guide key is not connected, the TV is not plugged in, the set-top box adb is dropped, the system hangs and does not respond, leading to mass failures), why Woolen cloth? Because this situation should be guaranteed by the tester (tool), adjust the correct physical environment to check and filter out, instead of repeatedly reporting it out and filtering it out by the developer again, To sum up the sentence: How does the test tool guarantee the test result? Robustness, purity! ! ! Especially in the early days when the system is unstable and there are many failures. It is difficult to not rely on manual intervention. It is still necessary for testers to monitor the operation status. |
| Mail Send | Current plan: Not done so far, copy and paste the “simple classification” into ICenter, write a test summary, arrange the failed item analysis task, and then forward the email. Why not automatically send the email? A: Still the above problem, we can’t confirm that the report is the report we want. If the spam data sent automatically is not necessary for analysis, it will increase the communication cost. Testers will frequently ask, can this result prove to be a system problem? How do I find many problems caused by the environment, are you sure this is a normal and robust test result? So the tester had to go back to confirm the test result again, retry again, and get a robust result consistent with experience |
Problem level: low Personal suggestion, automatic mail forwarding is not Necessary, because the tester needs to filter based on experience and designate a responsible person to arrange the analysis task after the test. Because the failure items are all domains, and the developers are divided into domains. If you don’t arrange it, each domain will be confused when facing a bunch of data. I don’t know which one to analyze. |
Remarks
Some supplements
The current architecture is
The win side is the console, and the win side needs to be the Linux compiler server and the Ubuntu test machine hanging Loaded as a network drive
The Ubuntu test machine needs dual network ports and connects to 10 and 192 external networks at the same time.
The set-top box is wired to 10, wireless to 192 external networks
Another consideration:
Can the win terminal be completely shielded, and all modules are moved to the Ubuntu test machine,
Ubuntu Compile the version on the remote compile server
Configure the Apache server on Ubuntu to force the upgrade of the set-top box.
Start the test directly on Ubuntu, retry
parse the results on Ubuntu, and back up the results on the server.
Then manually obtain the server backup and send an email Arrange analysis tasks.
The advantage of the above over the current solution is that it eliminates the need to remotely execute commands in Ubuntu on the win side, and the execution results standard output & log printing echoes on the win side is a troublesome thing does not echo Dehua as a control The win-end of the Taiwan is tasteless.