This would be a living document to capture various baselining snapshots while :-
Troubleshooting and applying a patch
Changing Software or Network configurations
Adding new scenarios and changing load share
Source Code: https://github.com/Bahmni/performance-test
...
Expand |
---|
title | 40 Concurrent Users - Configuration |
---|
|
Performance environment was running on AWS EKS Custer with single node Node (EC2: t3-large) RAM 8GB 2 vCPU 100GB Secondary storage AWS LINUX x86_64
Info |
---|
Total 20 application pods in cluster such as openmrs, bahmni-web, postgresql, abdm etc |
Database (AWS RDS service: db.t3.xlarge) RAM 16GB, 4 vCPU (2 core, 2.5 GHz Intel Scalable Processor) 100GB Secondary storage MySQL, max_connections = 1304
OpenMRS Tomcat - Server Code Block |
---|
Server version: Apache Tomcat/7.0.94
Server built: Apr 10 2019 16:56:40 UTC
Server number: 7.0.94.0
OS Name: Linux
OS Version: 5.4.204-113.362.amzn2.x86_64
Architecture: amd64
JVM Version: 1.8.0_212-8u212-b01-1~deb9u1-b01
ThreadPool: Max 200, Min 25 (Default server.xml) |
OpenMRS - Heap Initial Heap: 1024 MB Max Heap: 1536 MB
-Xms1024m -Xmx1536m -XX:NewSize=512m -XX:MaxNewSize=512m -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=1024m -XX:InitialCodeCacheSize=64m -XX:ReservedCodeCacheSize=96m -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=40 -XX:+UseParNewGC -XX:ParallelGCThreads=2 -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+CMSCompactWhenClearAllSoftRefs -XX:CMSInitiatingOccupancyFraction=85 -XX:+CMSScavengeBeforeRemark -XX:+UseGCOverheadLimit -XX:+UseStringDeduplication
Openmrs Connection Pooling Code Block |
---|
hibernate.c3p0.max_size=50
hibernate.c3p0.min_size=0
hibernate.c3p0.timeout=100
hibernate.c3p0.max_statements=0
hibernate.c3p0.idle_test_period=3000
hibernate.c3p0.acquire_increment=1 |
|
...
| | | | | | | |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 5760 | 152 | 484 | 648 | 1389 |
Existing Patient Search using ID → Start OPD Visit | 30% | 4320 | 54 | 472 | 676 | 1977 |
Existing Patient Search using Name → Start OPD Visit | 20% | 4320 | 119 | 352 | 507 | 1492 |
Upload Patient Document | 10% | 1440 | 142 | 482 | 581 | 1135 |
Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 5760 | 1364 | 4056 | 4531 | 7291 |
Expand |
---|
title | 70 Concurrent Users - Configuration |
---|
|
Performance environment was running on AWS EKS Custer with single node Node (EC2: m5-xlarge) RAM 16GB 4 vCPU 100GB Secondary storage AWS LINUX x86_64
Info |
---|
Total 20 application pods in cluster such as openmrs, bahmni-web, postgresql, abdm etc |
Database (AWS RDS service: db.t3.xlarge) RAM 16GB, 4 vCPU (2 core, 2.5 GHz Intel Scalable Processor) 100GB Secondary storage MySQL, max_connections = 1304
OpenMRS Tomcat - Server Code Block |
---|
Server version: Apache Tomcat/7.0.94
Server built: Apr 10 2019 16:56:40 UTC
Server number: 7.0.94.0
OS Name: Linux
OS Version: 5.4.204-113.362.amzn2.x86_64
Architecture: amd64
JVM Version: 1.8.0_212-8u212-b01-1~deb9u1-b01
ThreadPool: Max 200, Min 25 (Default server.xml) |
OpenMRS - Heap Initial Heap: 1024 MB Max Heap: 2536 MB
-Xms1024m -Xmx2536m -XX:NewSize=512m -XX:MaxNewSize=512m -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=1024m -XX:InitialCodeCacheSize=64m -XX:ReservedCodeCacheSize=96m -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=40 -XX:+UseParNewGC -XX:ParallelGCThreads=2 -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+CMSCompactWhenClearAllSoftRefs -XX:CMSInitiatingOccupancyFraction=85 -XX:+CMSScavengeBeforeRemark -XX:+UseGCOverheadLimit -XX:+UseStringDeduplication
Openmrs Connection Pooling Code Block |
---|
hibernate.c3p0.max_size=50
hibernate.c3p0.min_size=0
hibernate.c3p0.timeout=100
hibernate.c3p0.max_statements=0
hibernate.c3p0.idle_test_period=3000
hibernate.c3p0.acquire_increment=1 |
|
...
| | | | | | | |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 10080 | 130 | 252 | 309 | 674 |
Existing Patient Search using ID → Start OPD Visit | 30% | 7200 | 49 | 305 | 464 | 2660 |
Existing Patient Search using Name → Start OPD Visit | 20% | 7200 | 135 | 278 | 348 | 573 |
Upload Patient Document | 10% | 2160 | 111 | 206 | 253 | 459 |
Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 10080 | 998 | 2331 | 2608 | 4134 |
...
Note: The tests performed here are of most demand to users using Bahmni lite compared to the real world clinic activities. So it is assumed safe to implement for the suggested concurrent users under a cluster even when the maximum time response numbers for some activities are not optimal at the moment under test.
⭕️ Base Configuration
The below details are historical data observed during each tests with different configurations at the start of performance analysis for Bahmni lite. Kindly view them for data purposes and not as references.
Expand |
---|
|
Performance environment was running on AWS EKS Custer with single node Node (EC2: m5-xlarge) RAM 16GB 4 vCPU 100GB Secondary storage AWS LINUX x86_64
Info |
---|
Total 15 application pods in cluster such as openmrs, bahmni-web, postgresql, rabbitmq etc |
Database (AWS RDS service: db.t3.xlarge) RAM 16GB, 4 vCPU (2 core, 2.5 GHz Intel Scalable Processor) 100GB Secondary storage MySQL, max_connections = 1304
OpenMRS Tomcat - Server Code Block |
---|
Server version: Apache Tomcat/7.0.94
Server built: Apr 10 2019 16:56:40 UTC
Server number: 7.0.94.0
OS Name: Linux
OS Version: 5.4.204-113.362.amzn2.x86_64
Architecture: amd64
JVM Version: 1.8.0_212-8u212-b01-1~deb9u1-b01
ThreadPool: Max 200, Min 25 (Default server.xml) |
OpenMRS - Heap Initial Heap: 256 MB Max Heap: 768 MB
-Xms256m -Xmx768m -XX:PermSize=256m -XX:MaxPermSize=512m
Openmrs Connection Pooling Code Block |
---|
hibernate.c3p0.max_size=50
hibernate.c3p0.min_size=0
hibernate.c3p0.timeout=100
hibernate.c3p0.max_statements=0
hibernate.c3p0.idle_test_period=3000
hibernate.c3p0.acquire_increment=1 |
|
📙 10 Concurrent Users
Expand |
---|
|
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220911073147512_10users_30mins_768mb/index.html Report Observations: Simulations | Scenario | Load share | Patient Count | Min Time (ms) | Max Time (ms) |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 174 | 100 | 1077 | Existing Patient Search using ID → Start OPD Visit | 30% | 106 | 243 | 1550 | Existing Patient Search using Name → Start OPD Visit | 20% | 107 | 243 | 1437 | Upload Patient Document | 10% | 27 | 228 | 2169 | Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 187 | 185 | 2092 |
|
...
Expand |
---|
|
Note |
---|
Heap Size increased to 2GB since the base heap size wasn't enough to support more than 15 concurrent users |
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220912043950582_25users_15mins_2gb/index.html 🔰 Observations: Simulations | Scenario | Load share | Patient Count | Min Time (ms) | Max Time (ms) |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 86 | 161 | 1193 | Existing Patient Search using ID → Start OPD Visit | 30% | 74 | 320 | 1113 | Existing Patient Search using Name → Start OPD Visit | 20% | 64 | 310 | 1144 | Upload Patient Document | 10% | 21 | 213 | 843 | Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 107 | 259 | 846 |
|
...
Expand |
---|
|
Note |
---|
Network: 50 MBPS Heap Size: 2GB
|
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220912053731908_40users_15mins_2gb/index.html 🔰 Observations: Report Observations: Simulations | Scenario | Load share | Patient Count | Min Time (ms) | Max Time (ms) |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 130 | 162 | 1523 | Existing Patient Search using ID → Start OPD Visit | 30% | 109 | 356 | 2921 | Existing Patient Search using Name → Start OPD Visit | 20% | 124 | 337 | 4125 | Upload Patient Document | 10% | 36 | 238 | 579 | Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 142 | 252 | 1289 |
|
...
Expand |
---|
|
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220912071131867_70users_30mins_2gb/index.html 🔰 Observations: Report Observations: Simulations | Scenario | Load share | Patient Count | Min Time (ms) | Max Time (ms) | Notes |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 251 | 198 | 52939 | | Existing Patient Search using ID → Start OPD Visit | 30% | 175 | 1200 | 54692 | | Existing Patient Search using Name → Start OPD Visit | 20% | 162 | 979 | 54510 | | Upload Patient Document | 10% | 62 | 409 | 7848 | | Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 193 | 558 | 44887 | |
|
📙 90 Concurrent Users - Peak Traffic condition
Expand |
---|
|
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220912061644613_90users_30mins_8gb/index.html 🔰 Observations: OpenMRS Pod was consuming 11.5GB Memory and 3.22 CPU Overall Cluster was at 12.5GB RAM and 3.6 CPU More than 50% of the consultation and 95% all of Patient lab order fetch api calls suffered timeout.
Info |
---|
RDS showed total of 100+ connection during execution whereas OpenMRS Hibernate connection pool size is 50. Since the Memory and CPU consumption was equally high, we suspect that the application was busy fighting for establishing the connection concurrently. We need to identify a way to override OpenMRS connection pool configuration and rerun the tests. |
Report Observations: Simulations | Scenario | Load share | Patient Count | Min Time (ms) | Max Time (ms) | Notes |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 254 | 188 | 55802 | | Existing Patient Search using ID → Start OPD Visit | 30% | 192 | 1031 | 56250 | | Existing Patient Search using Name → Start OPD Visit | 20% | 177 | 651 | 53063 | | Upload Patient Document | 10% | 62 | 545 | 59070 | | Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 200 | 626 | 53271 | |
|
⭕️ Pace Based Framework
The framework is based on iterations per time type volumes i.e. a dedicated pace is set for each persona simulation.
📙 40 Concurrent Users - Standard Traffic condition
Expand |
---|
|
Duration: 30 mins Ramp Up: 3 mins
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220923121606971_40users_30mins_defaultJVM/index.html 🔰 Observations: Report Observations: Simulations | Scenario | Load share | Patient Count | Min Time (ms) | 95th Percentile (ms) | 99th Percentile (ms) | Max Time (ms) |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 108 | 88 | 191 | 328 | 378 | Existing Patient Search using ID → Start OPD Visit | 30% | 96 | 243 | 481 | 557 | 571 | Existing Patient Search using Name → Start OPD Visit | 20% | 87 | 243 | 579 | 694 | 749 | Upload Patient Document | 10% | 27 | 164 | 209 | 280 | 304 | Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 120 | 332 | 531 | 610 | 1153 |
|
...
Expand |
---|
|
Duration: 30 mins Ramp Up: 3 mins
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220923133122220_70users_30mins_defaultJVM/index.html 🔰 Observations: Report Observations: Simulations | Scenario | Load share | Patient Count | Min Time (ms) | 95th Percentile (ms) | 99th Percentile (ms) | Max Time (ms) |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 202 | 85 | 968 | 1058 | 1120 | Existing Patient Search using ID → Start OPD Visit | 30% | 163 | 232 | 1254 | 1700 | 1860 | Existing Patient Search using Name → Start OPD Visit | 20% | 143 | 241 | 1221 | 1630 | 1866 | Upload Patient Document | 10% | 40 | 144 | 984 | 1281 | 1338 | Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 210 | 329 | 1736 | 2538 | 2654 |
|
...
Expand |
---|
|
Duration: 30 mins Ramp Up: 3 mins
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220923141108441_90users_30mins_defaultJVM/index.html 🔰 Observations: Report Observations: Simulations | Scenario | Load share | Patient Count | Min Time (ms) | 95th Percentile (ms) | 99th Percentile (ms) | Max Time (ms) |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 274 | 81 | 995 | 1064 | 1843 | Existing Patient Search using ID → Start OPD Visit | 30% | 192 | 250 | 1820 | 2193 | 2410 | Existing Patient Search using Name → Start OPD Visit | 20% | 188 | 239 | 2109 | 2223 | 2465 | Upload Patient Document | 10% | 57 | 167 | 972 | 1366 | 1416 | Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 270 | 339 | 2656 | 3752 | 4149 |
|
The JVM was tuned for better efficiency from the default configuration
...
Expand |
---|
|
Duration: 30 mins Ramp Up: 3 mins
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220927073846406_40users_30mins_tunedJVM/index.html 🔰 Observations: Report Observations: Simulations | Scenario | Load share | Patient Count | Min Time (ms) | 95th Percentile (ms) | 99th Percentile (ms) | Max Time (ms) |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 119 | 154 | 294 | 361 | 544 | Existing Patient Search using ID → Start OPD Visit | 30% | 75 | 242 | 408 | 453 | 463 | Existing Patient Search using Name → Start OPD Visit | 20% | 96 | 270 | 612 | 765 | 766 | Upload Patient Document | 10% | 35 | 138 | 348 | 474 | 504 | Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 120 | 490 | 687 | 769 | 778 |
|
...
Expand |
---|
|
Duration: 30 mins Ramp Up: 3 mins
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220927100508288_70users_30mins_tunedJVM/index.html 🔰 Observations: Report Observations: Simulations | Scenario | Load share | Patient Count | Min Time (ms) | 95th Percentile (ms) | 99th Percentile (ms) | Max Time (ms) |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 206 | 164 | 366 | 537 | 677 | Existing Patient Search using ID → Start OPD Visit | 30% | 146 | 249 | 661 | 806 | 894 | Existing Patient Search using Name → Start OPD Visit | 20% | 157 | 238 | 654 | 743 | 767 | Upload Patient Document | 10% | 60 | 132 | 600 | 710 | 710 | Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 210 | 497 | 978 | 1075 | 1236 |
|
...
Expand |
---|
|
Duration: 30 mins Ramp Up: 3 mins
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220927092902911_90users_30mins_tunedJVM/index.html 🔰 Observations: Report Observations: Simulations | Scenario | Load share | Patient Count | Min Time (ms) | 95th Percentile (ms) | 99th Percentile (ms) | Max Time (ms) |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 260 | 153 | 362 | 423 | 526 | Existing Patient Search using ID → Start OPD Visit | 30% | 199 | 268 | 786 | 960 | 1163 | Existing Patient Search using Name → Start OPD Visit | 20% | 197 | 276 | 844 | 952 | 1120 | Upload Patient Document | 10% | 62 | 152 | 537 | 741 | 860 | Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 270 | 511 | 1037 | 1321 | 1551 |
|
...