This would be a living document to capture various baselining snapshots while :-
Troubleshooting and applying a patch
Changing Software or Network configurations
Adding new scenarios and changing load share
Source Code: https://github.com/Bahmni/performance-test
40 Concurrent Users - Configuration
Performance environment was running on AWS EKS Custer with single node
Node (EC2: t3-large)
RAM 8GB
2 vCPU
100GB Secondary storage
AWS LINUX x86_64
Database (AWS RDS service: db.t3.xlarge)
RAM 16GB,
4 vCPU (2 core, 2.5 GHz Intel Scalable Processor)
100GB Secondary storage
MySQL, max_connections = 1304
OpenMRS Tomcat - Server
Server version: Apache Tomcat/7.0.94
Server built: Apr 10 2019 16:56:40 UTC
Server number: 7.0.94.0
OS Name: Linux
OS Version: 5.4.204-113.362.amzn2.x86_64
Architecture: amd64
JVM Version: 1.8.0_212-8u212-b01-1~deb9u1-b01
ThreadPool: Max 200, Min 25 (Default server.xml)
OpenMRS - Heap
Initial Heap: 1024 MB
Max Heap: 1536 MB
-Xms1024m -Xmx1536m -XX:NewSize=512m -XX:MaxNewSize=512m -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=1024m -XX:InitialCodeCacheSize=64m -XX:ReservedCodeCacheSize=96m -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=40 -XX:+UseParNewGC -XX:ParallelGCThreads=2 -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+CMSCompactWhenClearAllSoftRefs -XX:CMSInitiatingOccupancyFraction=85 -XX:+CMSScavengeBeforeRemark -XX:+UseGCOverheadLimit -XX:+UseStringDeduplication
Openmrs Connection Pooling
hibernate.c3p0.max_size=50
hibernate.c3p0.min_size=0
hibernate.c3p0.timeout=100
hibernate.c3p0.max_statements=0
hibernate.c3p0.idle_test_period=3000
hibernate.c3p0.acquire_increment=1
Report Link: https://bahmni.github.io/performance-test/longduration_report-20230130141239257_40users_24hrs_all_omods_afterhipfix/index.html
Report Observations:
| | | | | | | |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 5760 | 152 | 484 | 648 | 1389 |
Existing Patient Search using ID → Start OPD Visit | 30% | 4320 | 54 | 472 | 676 | 1977 |
Existing Patient Search using Name → Start OPD Visit | 20% | 4320 | 119 | 352 | 507 | 1492 |
Upload Patient Document | 10% | 1440 | 142 | 482 | 581 | 1135 |
Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 5760 | 1364 | 4056 | 4531 | 7291 |
70 Concurrent Users - Configuration
Performance environment was running on AWS EKS Custer with single node
Node (EC2: m5-xlarge)
RAM 16GB
4 vCPU
100GB Secondary storage
AWS LINUX x86_64
Database (AWS RDS service: db.t3.xlarge)
RAM 16GB,
4 vCPU (2 core, 2.5 GHz Intel Scalable Processor)
100GB Secondary storage
MySQL, max_connections = 1304
OpenMRS Tomcat - Server
Server version: Apache Tomcat/7.0.94
Server built: Apr 10 2019 16:56:40 UTC
Server number: 7.0.94.0
OS Name: Linux
OS Version: 5.4.204-113.362.amzn2.x86_64
Architecture: amd64
JVM Version: 1.8.0_212-8u212-b01-1~deb9u1-b01
ThreadPool: Max 200, Min 25 (Default server.xml)
OpenMRS - Heap
Initial Heap: 1024 MB
Max Heap: 2536 MB
-Xms1024m -Xmx2536m -XX:NewSize=512m -XX:MaxNewSize=512m -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=1024m -XX:InitialCodeCacheSize=64m -XX:ReservedCodeCacheSize=96m -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=40 -XX:+UseParNewGC -XX:ParallelGCThreads=2 -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+CMSCompactWhenClearAllSoftRefs -XX:CMSInitiatingOccupancyFraction=85 -XX:+CMSScavengeBeforeRemark -XX:+UseGCOverheadLimit -XX:+UseStringDeduplication
Openmrs Connection Pooling
hibernate.c3p0.max_size=50
hibernate.c3p0.min_size=0
hibernate.c3p0.timeout=100
hibernate.c3p0.max_statements=0
hibernate.c3p0.idle_test_period=3000
hibernate.c3p0.acquire_increment=1
Report Link: https://bahmni.github.io/performance-test/longduration_report-20230213133118638_70users_24hours_AfterHIPfix_m5xlarge/index.html
Report Observations:
| | | | | | | |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 10080 | 130 | 252 | 309 | 674 |
Existing Patient Search using ID → Start OPD Visit | 30% | 7200 | 49 | 305 | 464 | 2660 |
Existing Patient Search using Name → Start OPD Visit | 20% | 7200 | 135 | 278 | 348 | 573 |
Upload Patient Document | 10% | 2160 | 111 | 206 | 253 | 459 |
Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 10080 | 998 | 2331 | 2608 | 4134 |
🔰 Observations:
Note: The tests performed here are of most demand to users using Bahmni lite compared to the real world clinic activities. So it is assumed safe to implement for the suggested concurrent users under a cluster even when the maximum time response numbers for some activities are not optimal at the moment under test.
The below details are historical data observed during each tests with different configurations at the start of performance analysis for Bahmni lite. Kindly view them for data purposes and not as references.
Details
Performance environment was running on AWS EKS Custer with single node
Node (EC2: m5-xlarge)
RAM 16GB
4 vCPU
100GB Secondary storage
AWS LINUX x86_64
Database (AWS RDS service: db.t3.xlarge)
RAM 16GB,
4 vCPU (2 core, 2.5 GHz Intel Scalable Processor)
100GB Secondary storage
MySQL, max_connections = 1304
OpenMRS Tomcat - Server
Server version: Apache Tomcat/7.0.94
Server built: Apr 10 2019 16:56:40 UTC
Server number: 7.0.94.0
OS Name: Linux
OS Version: 5.4.204-113.362.amzn2.x86_64
Architecture: amd64
JVM Version: 1.8.0_212-8u212-b01-1~deb9u1-b01
ThreadPool: Max 200, Min 25 (Default server.xml)
OpenMRS - Heap
Initial Heap: 256 MB
Max Heap: 768 MB
-Xms256m -Xmx768m -XX:PermSize=256m -XX:MaxPermSize=512m
Openmrs Connection Pooling
hibernate.c3p0.max_size=50
hibernate.c3p0.min_size=0
hibernate.c3p0.timeout=100
hibernate.c3p0.max_statements=0
hibernate.c3p0.idle_test_period=3000
hibernate.c3p0.acquire_increment=1
Details
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220911073147512_10users_30mins_768mb/index.html
Report Observations:
Simulations | Scenario | Load share | Patient Count | Min Time (ms) | Max Time (ms) |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 174 | 100 | 1077 |
Existing Patient Search using ID → Start OPD Visit | 30% | 106 | 243 | 1550 |
Existing Patient Search using Name → Start OPD Visit | 20% | 107 | 243 | 1437 |
Upload Patient Document | 10% | 27 | 228 | 2169 |
Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 187 | 185 | 2092 |
Details
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220912043950582_25users_15mins_2gb/index.html
🔰 Observations:
Simulations | Scenario | Load share | Patient Count | Min Time (ms) | Max Time (ms) |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 86 | 161 | 1193 |
Existing Patient Search using ID → Start OPD Visit | 30% | 74 | 320 | 1113 |
Existing Patient Search using Name → Start OPD Visit | 20% | 64 | 310 | 1144 |
Upload Patient Document | 10% | 21 | 213 | 843 |
Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 107 | 259 | 846 |
Details
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220912053731908_40users_15mins_2gb/index.html
🔰 Observations:
Report Observations:
Simulations | Scenario | Load share | Patient Count | Min Time (ms) | Max Time (ms) |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 130 | 162 | 1523 |
Existing Patient Search using ID → Start OPD Visit | 30% | 109 | 356 | 2921 |
Existing Patient Search using Name → Start OPD Visit | 20% | 124 | 337 | 4125 |
Upload Patient Document | 10% | 36 | 238 | 579 |
Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 142 | 252 | 1289 |
Details
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220912071131867_70users_30mins_2gb/index.html
🔰 Observations:
Report Observations:
Simulations | Scenario | Load share | Patient Count | Min Time (ms) | Max Time (ms) | Notes |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 251 | 198 | 52939 | |
Existing Patient Search using ID → Start OPD Visit | 30% | 175 | 1200 | 54692 | |
Existing Patient Search using Name → Start OPD Visit | 20% | 162 | 979 | 54510 | |
Upload Patient Document | 10% | 62 | 409 | 7848 | |
Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 193 | 558 | 44887 | |
Details
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220912061644613_90users_30mins_8gb/index.html
🔰 Observations:
OpenMRS Pod was consuming 11.5GB Memory and 3.22 CPU
Overall Cluster was at 12.5GB RAM and 3.6 CPU
More than 50% of the consultation and 95% all of Patient lab order fetch api calls suffered timeout.
Report Observations:
Simulations | Scenario | Load share | Patient Count | Min Time (ms) | Max Time (ms) | Notes |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 254 | 188 | 55802 | |
Existing Patient Search using ID → Start OPD Visit | 30% | 192 | 1031 | 56250 | |
Existing Patient Search using Name → Start OPD Visit | 20% | 177 | 651 | 53063 | |
Upload Patient Document | 10% | 62 | 545 | 59070 | |
Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 200 | 626 | 53271 | |
The framework is based on iterations per time type volumes i.e. a dedicated pace is set for each persona simulation.
Details
Duration: 30 mins
Ramp Up: 3 mins
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220923121606971_40users_30mins_defaultJVM/index.html
🔰 Observations:
Report Observations:
Simulations | Scenario | Load share | Patient Count | Min Time (ms) | 95th Percentile (ms) | 99th Percentile (ms) | Max Time (ms) |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 108 | 88 | 191 | 328 | 378 |
Existing Patient Search using ID → Start OPD Visit | 30% | 96 | 243 | 481 | 557 | 571 |
Existing Patient Search using Name → Start OPD Visit | 20% | 87 | 243 | 579 | 694 | 749 |
Upload Patient Document | 10% | 27 | 164 | 209 | 280 | 304 |
Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 120 | 332 | 531 | 610 | 1153 |
Details
Duration: 30 mins
Ramp Up: 3 mins
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220923133122220_70users_30mins_defaultJVM/index.html
🔰 Observations:
Report Observations:
Simulations | Scenario | Load share | Patient Count | Min Time (ms) | 95th Percentile (ms) | 99th Percentile (ms) | Max Time (ms) |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 202 | 85 | 968 | 1058 | 1120 |
Existing Patient Search using ID → Start OPD Visit | 30% | 163 | 232 | 1254 | 1700 | 1860 |
Existing Patient Search using Name → Start OPD Visit | 20% | 143 | 241 | 1221 | 1630 | 1866 |
Upload Patient Document | 10% | 40 | 144 | 984 | 1281 | 1338 |
Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 210 | 329 | 1736 | 2538 | 2654 |
Details
Duration: 30 mins
Ramp Up: 3 mins
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220923141108441_90users_30mins_defaultJVM/index.html
🔰 Observations:
Report Observations:
Simulations | Scenario | Load share | Patient Count | Min Time (ms) | 95th Percentile (ms) | 99th Percentile (ms) | Max Time (ms) |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 274 | 81 | 995 | 1064 | 1843 |
Existing Patient Search using ID → Start OPD Visit | 30% | 192 | 250 | 1820 | 2193 | 2410 |
Existing Patient Search using Name → Start OPD Visit | 20% | 188 | 239 | 2109 | 2223 | 2465 |
Upload Patient Document | 10% | 57 | 167 | 972 | 1366 | 1416 |
Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 270 | 339 | 2656 | 3752 | 4149 |
The JVM was tuned for better efficiency from the default configuration
Configuration
-Xms768m -Xmx768m -XX:NewSize=384m -XX:MaxNewSize=384m -XX:MetaspaceSize=768m -XX:MaxMetaspaceSize=768m -XX:InitialCodeCacheSize=64m -XX:ReservedCodeCacheSize=96m -XX:SurvivorRatio=16 -XX:TargetSurvivorRatio=50 -XX:MaxTenuringThreshold=15 -XX:+UseParNewGC -XX:ParallelGCThreads=16 -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+CMSCompactWhenClearAllSoftRefs -XX:CMSInitiatingOccupancyFraction=85 -XX:+CMSScavengeBeforeRemark
Details
Duration: 30 mins
Ramp Up: 3 mins
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220927073846406_40users_30mins_tunedJVM/index.html
🔰 Observations:
Report Observations:
Simulations | Scenario | Load share | Patient Count | Min Time (ms) | 95th Percentile (ms) | 99th Percentile (ms) | Max Time (ms) |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 119 | 154 | 294 | 361 | 544 |
Existing Patient Search using ID → Start OPD Visit | 30% | 75 | 242 | 408 | 453 | 463 |
Existing Patient Search using Name → Start OPD Visit | 20% | 96 | 270 | 612 | 765 | 766 |
Upload Patient Document | 10% | 35 | 138 | 348 | 474 | 504 |
Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 120 | 490 | 687 | 769 | 778 |
Details
Duration: 30 mins
Ramp Up: 3 mins
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220927100508288_70users_30mins_tunedJVM/index.html
🔰 Observations:
Report Observations:
Simulations | Scenario | Load share | Patient Count | Min Time (ms) | 95th Percentile (ms) | 99th Percentile (ms) | Max Time (ms) |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 206 | 164 | 366 | 537 | 677 |
Existing Patient Search using ID → Start OPD Visit | 30% | 146 | 249 | 661 | 806 | 894 |
Existing Patient Search using Name → Start OPD Visit | 20% | 157 | 238 | 654 | 743 | 767 |
Upload Patient Document | 10% | 60 | 132 | 600 | 710 | 710 |
Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 210 | 497 | 978 | 1075 | 1236 |
Details
Duration: 30 mins
Ramp Up: 3 mins
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220927092902911_90users_30mins_tunedJVM/index.html
🔰 Observations:
Report Observations:
Simulations | Scenario | Load share | Patient Count | Min Time (ms) | 95th Percentile (ms) | 99th Percentile (ms) | Max Time (ms) |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 260 | 153 | 362 | 423 | 526 |
Existing Patient Search using ID → Start OPD Visit | 30% | 199 | 268 | 786 | 960 | 1163 |
Existing Patient Search using Name → Start OPD Visit | 20% | 197 | 276 | 844 | 952 | 1120 |
Upload Patient Document | 10% | 62 | 152 | 537 | 741 | 860 |
Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 270 | 511 | 1037 | 1321 | 1551 |
The scenario based pace framework enables various pace distribution among different scenarios individually.
Configuration
-Xms768m -Xmx768m -XX:NewSize=384m -XX:MaxNewSize=384m -XX:MetaspaceSize=768m -XX:MaxMetaspaceSize=768m -XX:InitialCodeCacheSize=64m -XX:ReservedCodeCacheSize=96m -XX:SurvivorRatio=16 -XX:TargetSurvivorRatio=50 -XX:MaxTenuringThreshold=15 -XX:+UseParNewGC -XX:ParallelGCThreads=16 -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+CMSCompactWhenClearAllSoftRefs -XX:CMSInitiatingOccupancyFraction=85 -XX:+CMSScavengeBeforeRemark