On Thu, May 9, 2024 at 12:44 PM Gavin McKee via discuss
<ovs-discuss@openvswitch.org> wrote:
>
> I think a big part of the problem is the Mac_Binding updates, these
> are pinning OVN controll CPU to 100%, this is what leads to the main
> thread blocking the DNS lookup.
>
> Is there a way to stop these mac_binding updates?
This doesn't look good. Let me check and come back if it's possible
to disable mac binding timestamp updates or to increase mac binding
ageing limit.
Numan
>
>
> 2024-05-09T00:25:33.588Z|00690|jsonrpc|DBG|tcp:10.193.1.3:6642:
> received notification, method="update3",
> params=[["monid","OVN_Southbound"],"ecb00de5-bd2e-4238-bcbc-60e0f6c64ba9",{"MAC_Binding":{"cdd0fb6b-428f-461e-9b7c-0ecd12ab7634":{"modify":{"timestamp":1715214333315}},"b110efd1-998e-40dd-906c-de8d260747b4":{"modify":{"timestamp":1715214333315}},"13f1ce55-0c43-46ba-ae59-ae1c0a414165":{"modify":{"timestamp":1715214333315}},"f866fc8e-6f43-4e4d-9ec7-6bfa3845d150":{"modify":{"timestamp":1715214333315}},"792507cc-dba7-4a3a-90ed-53cffcf5e915":{"modify":{"timestamp":1715214333315}},"140d0d21-7f6d-496a-9e66-2b4c6dc93787":{"modify":{"timestamp":1715214333315}},"5608683e-7342-4885-9a4e-6aac8f015464":{"modify":{"timestamp":1715214333315}},"43a9515f-f270-4275-be11-8734ea6b790f":{"modif
> y":{"timestamp":1715214333315}},"67c87621-9321-4000-86b2-a9117b803cfe":{"modify":{"timestamp":1715214333315}},"238c7da1-992b-47cc-9026-f636682a2c50":{"modify":{"timestamp":1715214333315}},"af9b65da-37a9-45e8-80f4-f0e1fb43bee2":{"modify":{"timestamp":1715214333315}},"5d38a24a-e4da-4595-b3e9-9029d21f8be9":{"modify":{"timestamp":1715214333315}},"7718ba49-a8b3-4abb-bf69-33502bb832ea":{"modify":{"timestamp":1715214333315}},"6569a978-2d5f-4177-a1cd-a04c4d3ae0cb":{"modify":{"timestamp":1715214333315}},"b48d9451-e62a-49d0-b589-a12b53c8f27d":{"modify":{"timestamp":1715214333315}},"6b133fc5-536f-425a-aa3d-092c805f93c6":{"modify":{"timestamp":1715214333315}},"baebb40b-a75b-4be8-b645-f57d5ac5d586":{"modify":{"timestamp":1715214333315}},"95fb8300-4c3d-4451-9520-e16a156724aa":{"modify":{"ti
> mestamp":1715214333315}},"e941d3c8-be34-496e-a5d3-c54f5b1ecb60":{"modify":{"timestamp":1715214333315}},"3e481826-753a-49b2-ac77-1db3b4a87cb6":{"modify":{"timestamp":1715214333315}},"09e110cb-18ca-44ee-bc49-6f9068acc8a1":{"modify":{"timestamp":1715214333315}},"b28ff12f-05c4-4299-ab4e-08cea3cddff9":{"modify":{"timestamp":1715214333315}},"9e53ccf8-ee50-4cf8-972a-3fa3d3ea01af":{"modify":{"timestamp":1715214333315}},"52dba98a-a66e-4f98-9c98-e040291cfb75":{"modify":{"timestamp":1715214333315}},"356f5097-98cc-4904-9150-43c8af59e08e":{"modify":{"timestamp":1715214333315}},"cd77807b-0cbc-479a-bc15-b514e9731e00":{"modify":{"timestamp":1715214333315}},"a933f26c-edcf-4817-9477-0ff5c696ea43":{"modify":{"timestamp":1715214333315}},"9b4122f7-6976-46b7-823c-f25085b807b9":{"modify":{"timestamp
> ":1715214333315}},"4aa1aaaa-93ca-4977-9461-2a910b9c5cda":{"modify":{"timestamp":1715214333315}},"e4289076-60dc-435b-9ca7-4f3ae61c5f3c":{"modify":{"timestamp":1715214333315}},"eb43b529-7b31-451a-96bb-27bdcc780428":{"modify":{"timestamp":1715214333315}},"302449cd-a8c6-45fe-8325-f63778fad1f0":{"modify":{"timestamp":1715214333315}},"38fe8afc-65b8-4af5-a29f-6f26a27fd2a4":{"modify":{"timestamp":1715214333315}},"3d8c2983-9f97-4b0b-a92c-5c3a2d30c270":{"modify":{"timestamp":1715214333315}},"45a4eb3e-9ccf-4c2d-ace3-0d099dd0cbb0":{"modify":{"timestamp":1715214333315}},"dd6cb224-bf05-4947-9d65-8e5b6aebb233":{"modify":{"timestamp":1715214333315}},"c8eb33f2-5620-4021-8276-3ac6d9b5e06b":{"modify":{"timestamp":1715214333315}},"7e0086a9-03e1-4296-90ea-9de704cb2e6c":{"modify":{"timestamp":17152
> 14333315}},"4e8e9e74-1fc9-4d01-9e14-21dbf7fb6806":{"modify":{"timestamp":1715214333315}},"f0056172-519f-4398-9e10-cb1e37ceb27e":{"modify":{"timestamp":1715214333315}},"50775138-5237-4bcc-a61a-f802aac00ae9":{"modify":{"timestamp":1715214333315}},"78058de1-d87d-4e89-a842-c5502b6b39cd":{"modify":{"timestamp":1715214333315}},"da6916ad-1954-4bc9-93a7-2e8ee76dfe98":{"modify":{"timestamp":1715214333315}},"0d3212ce-3d9e-40af-8bb4-cc56a66a013c":{"modify":{"timestamp":1715214333315}},"9853cc65-3fc6-4437-843c-f42baf7c2f20":{"modify":{"timestamp":1715214333315}},"ca314888-a76f-4119-982e-04451796e2ed":{"modify":{"timestamp":1715214333315}},"bc9c8584-5026-44d5-89ee-c8452ddb0742":{"modify":{"timestamp":1715214333315}},"5eb7547e-56ab-4b29-94a2-443dfed288a9":{"modify":{"timestamp":171521433331
> 5}},"e33ebef7-6277-4f8d-9df0-9360bb83f851":{"modify":{"timestamp":1715214333315}},"7ed708f2-f977-43e6-a323-5ffdbb7df629":{"modify":{"timestamp":1715214333315}},"bc098ad3-0377-4d92-9231-5143795475b2":{"modify":{"timestamp":1715214333315}},"cec2d293-5ac9-4b17-8b4c-bcbb29df0ac3":{"modify":{"timestamp":1715214333315}},"883a3d3c-f05f-46b0-bf77-298118bcbae0":{"modify":{"timestamp":1715214333315}},"c460dc4e-411b-4749-bd8f-67e58a946d0e":{"modify":{"timestamp":1715214333315}},"2f8cb1f8-e00e-4f66-832a-fc5293784ba3":{"modify":{"timestamp":1715214333315}},"4a76bb0e-587f-4d4f-8325-68b816109b66":{"modify":{"timestamp":1715214333315}},"602ea4c6-6009-4a95-9941-6714dd7a22fd":{"modify":{"timestamp":1715214333315}},"204ea460-0e52-4da8-b4de-1518eb687c2a":{"modify":{"timestamp":1715214333315}},"ea
> abb659-b861-4e9e-a47a-39638dfd9577":{"modify":{"timestamp":1715214333315}},"6695541e-faac-40a8-ae54-e5d332282bfa":{"modify":{"timestamp":1715214333315}},"c40f4162-41fa-4bd6-bfbc-8a39694bcbfe":{"modify":{"timestamp":1715214333315}},"eab0b287-04c5-4175-bbc6-a5d577aaf2f2":{"modify":{"timestamp":1715214333315}},"0acfeaf0-6301-48de-af43-a7d5dfd2ffa4":{"modify":{"timestamp":1715214333315}},"0788daee-1e98-4983-a671-d2af98be9468":{"modify":{"timestamp":1715214333315}},"1f0a7c1d-10b1-4eb6-93ce-594e320cdddd":{"modify":{"timestamp":1715214333315}},"4de9b641-c942-4a71-a87b-487fe78c3843":{"modify":{"timestamp":1715214333315}},"c0489f74-3934-4f4e-ad45-f43ec5e97ad9":{"modify":{"timestamp":1715214333315}},"99aef5f9-76a0-4408-b9db-000da4b97758":{"modify":{"timestamp":1715214333315}},"91ab15b9-
> ea6b-41fe-99d7-4cee250fe31d":{"modify":{"timestamp":1715214333315}},"b6672f0c-1327-44d7-956f-aa93c32f9f6f":{"modify":{"timestamp":1715214333315}},"7d8e0611-3731-4014-b06f-5de9d04f8031":{"modify":{"timestamp":1715214333315}},"dabef001-f78f-4858-8121-6dc4fda38b29":{"modify":{"timestamp":1715214333315}},"5c346bd8-8e48-4cd7-b5b9-bac81f355fca":{"modify":{"timestamp":1715214333315}},"98aa2262-ec5f-4aca-96fa-0042680b02d8":{"modify":{"timestamp":1715214333315}},"c7f95e5c-f367-442c-849b-785f864e7321":{"modify":{"timestamp":1715214333315}},"50fd8169-f326-4527-8247-e5eb88ac6518":{"modify":{"timestamp":1715214333315}},"8bb6a483-0182-4347-a7e5-9c881a6cc9f8":{"modify":{"timestamp":1715214333315}},"4a161b62-1648-4f75-a03a-9359dd282c28":{"modify":{"timestamp":1715214333315}},"563e6e16-6caa-40
> e6-99c8-777032e8a7b4":{"modify":{"timestamp":1715214333315}},"addf3fd6-3ab0-4ce5-b994-4bad1c13e861":{"modify":{"timestamp":1715214333315}},"2fcaec6a-353c-4d6f-8795-f48851978559":{"modify":{"timestamp":1715214333315}},"840e1c4f-3342-4134-a7a3-1a4ceffa8324":{"modify":{"timestamp":1715214333315}},"34587b9f-1202-4950-8c16-8df1f5a6af08":{"modify":{"timestamp":1715214333315}},"153a228d-3eba-4fdb-bb7d-6c173c2a85eb":{"modify":{"timestamp":1715214333315}},"395dbe05-8400-4b82-90f9-4124727c56b0":{"modify":{"timestamp":1715214333315}},"acf70153-98b3-4b22-8fc3-a6446a28d82d":{"modify":{"timestamp":1715214333315}},"e495b4e2-dacb-494c-a199-935643e254ef":{"modify":{"timestamp":1715214333315}},"87da99be-6ffd-46a6-a81d-71f2f7d2ba6a":{"modify":{"timestamp":1715214333315}},"2b35ac1a-2af2-4145-9445
> -25c25728e71f":{"modify":{"timestamp":1715214333315}},"56a124c7-279e-4863-b383-b4b9d9578f87":{"modify":{"timestamp":1715214333315}},"818e32fd-3c09-4c47-95a5-edb249e93691":{"modify":{"timestamp":1715214333315}},"843f159d-5c03-4501-a507-6def83b4fa0e":{"modify":{"timestamp":1715214333315}},"6f9fe9ec-9c66-4243-abe0-804b881b3b9f":{"modify":{"timestamp":1715214333315}},"a4958493-b503-47bc-96db-6767459dafd8":{"modify":{"timestamp":1715214333315}},"eb32106a-91e5-478b-8d45-84d17edc1a5a":{"modify":{"timestamp":1715214333315}},"8249e2de-aa43-4fae-8537-514b4a148e50":{"modify":{"timestamp":1715214333315}},"0cb01f1d-2aeb-492e-80bc-f3cbb1886ee2":{"modify":{"timestamp":1715214333315}},"ff3e4c52-f695-4373-ac52-3cebf6abeef5":{"modify":{"timestamp":1715214333315}},"5dba1702-09dc-4a02-9c99-0efc65
> a483b1":{"modify":{"timestamp":1715214333315}},"9a8bb58e-f79a-4258-861a-1ee4f3361ce1":{"modify":{"timestamp":1715214333315}},"cfa1de91-5752-4314-85df-ecdd879fde82":{"modify":{"timestamp":1715214333315}},"1166c63f-72a8-4a0c-be46-7bb621402737":{"modify":{"timestamp":1715214333315}},"0b9e0218-937d-462a-b884-d3d4d046dd8e":{"modify":{"timestamp":1715214333315}},"0bb4f41c-e066-45c8-8835-959c6ae5e02d":{"modify":{"timestamp":1715214333315}},"8ed5a7ff-169c-40d9-869b-278296b530a2":{"modify":{"timestamp":1715214333315}},"23665810-125a-464b-8dd5-0854f0c40c66":{"modify":{"timestamp":1715214333315}},"01b290e6-d9fa-44f8-9715-194798e812d1":{"modify":{"timestamp":1715214333315}},"0f9bd36b-e3d0-40d7-9b3b-a6ed7adf3eac":{"modify":{"timestamp":1715214333315}},"823bddf3-150a-4768-963a-4fc8398a32dd"
> :{"modify":{"timestamp":1715214333315}},"ab643093-74b4-4497-9656-cc55b2934124":{"modify":{"timestamp":1715214333315}},"c8bfe712-30e4-4243-b1e5-6696566369fb":{"modify":{"timestamp":1715214333315}},"84a78c52-e9ff-4e48-90f2-481ecb23b0dd":{"modify":{"timestamp":1715214333315}},"07ea48c0-a50b-4254-ba8c-4aaa78ba0318":{"modify":{"timestamp":1715214333315}},"4ac13fd3-2006-4b3a-9174-67f216d1ef4b":{"modify":{"timestamp":1715214333315}},"9e54de53-22fd-47d9-85ac-4a630d4094da":{"modify":{"timestamp":1715214333315}},"d02e6f9b-48b3-4da8-a4c0-8b685f2d64e7":{"modify":{"timestamp":1715214333315}},"a3fec609-deaa-48b5-9afc-61c503a92789":{"modify":{"timestamp":1715214333315}},"83268cff-9509-4e05-be89-908c5ac816ee":{"modify":{"timestamp":1715214333315}},"e57eae45-59ee-4a45-b42a-d91d210ec8a9":{"modi
> fy":{"timestamp":1715214333315}},"ffad3f72-1e7e-47e9-93b3-4c22230456f4":{"modify":{"timestamp":1715214333315}},"6867760d-1ce3-414e-8993-3078d7885995":{"modify":{"timestamp":1715214333315}},"7f87e742-6590-4aee-b6e8-a2692a7289b2":{"modify":{"timestamp":1715214333315}},"27083ab3-c494-41ef-8b60-0c16d54ee3c5":{"modify":{"timestamp":1715214333315}},"16555c06-4962-4938-aa0f-69119bbf2669":{"modify":{"timestamp":1715214333315}},"7b6bfe37-da68-4c8c-8009-cf2dd50d8d0d":{"modify":{"timestamp":1715214333315}},"bc7882f1-4b74-4981-ba29-e05544aeec06":{"modify":{"timestamp":1715214333315}},"15335712-e293-45d8-8879-55f2b17aa5c0":{"modify":{"timestamp":1715214333315}},"5515bb75-0a26-4ebd-bb0a-3b8afca6f41e":{"modify":{"timestamp":1715214333315}},"26024163-90b3-401d-bc00-d04c8eb9ef31":{"modify":{"t
> imestamp":1715214333315}},"d0ee873c-8905-47e9-adbd-208974412905":{"modify":{"timestamp":1715214333315}},"826280a7-0d80-4dcb-ab39-54aa8098b2aa":{"modify":{"timestamp":1715214333315}},"cec9e9ad-4a39-4014-bef1-94cb77acf35f":{"modify":{"timestamp":1715214333315}},"1478cd42-1337-46a7-b40c-1c97615f7ede":{"modify":{"timestamp":1715214333315}},"de78ac43-3d08-450c-8c58-2d012beecbc6":{"modify":{"timestamp":1715214333315}},"ddd247bf-900d-4ae9-921a-a4b68efbdac7":{"modify":{"timestamp":1715214333315}},"63a694ec-c610-425d-84d0-cd951c571e73":{"modify":{"timestamp":1715214333315}},"7277e512-22d7-449e-9f24-b34997533c7f":{"modify":{"timestamp":1715214333315}},"65bd6fa3-e566-4ed0-ace6-36cc193cb1e3":{"modify":{"timestamp":1715214333315}},"1e1b7fb8-e229-433c-9637-0d08f82ed30d":{"modify":{"timestam
> p":1715214333315}},"3bfdcd54-929c-46c0-8504-35ca1bd9f151":{"modify":{"timestamp":1715214333315}},"473a3087-fe23-492b-948a-64485bb05bfa":{"modify":{"timestamp":1715214333315}},"cb3a530e-1ab7-41d1-94b5-3b1ba8d81a8e":{"modify":{"timestamp":1715214333315}},"83a4ae3b-23e7-4156-b7a3-624971dc2585":{"modify":{"timestamp":1715214333315}},"5ffa8602-6b7d-449a-9c7a-71394282ea72":{"modify":{"timestamp":1715214333315}},"bf1f048a-aa17-450e-81f7-866f13726753":{"modify":{"timestamp":1715214333315}},"4256652e-ecf7-4e51-92ea-06014742b3f8":{"modify":{"timestamp":1715214333315}},"a80a445c-9fd7-48b3-b3af-1b609aa57596":{"modify":{"timestamp":1715214333315}},"f5ddc5bf-d631-4759-bc31-7612e85c093d":{"modify":{"timestamp":1715214333315}},"cbea358f-3ea0-4bb0-8bf1-69e0421f0176":{"modify":{"timestamp":1715
> 214333315}},"38770eec-c48e-482d-9ce4-c24a3383b823":{"modify":{"timestamp":1715214333315}},"44f713b2-5b2d-44a7-a16a-09ff13ad51a5":{"modify":{"timestamp":1715214333315}},"8ad84c55-2a2f-4766-a97b-f83ddffa2657":{"modify":{"timestamp":1715214333315}},"c3ee1088-9c52-4044-b6ff-c07c7240970f":{"modify":{"timestamp":1715214333315}},"ccfebfc4-3821-42b3-bacd-919a3d930d05":{"modify":{"timestamp":1715214333315}},"f63b9138-1c74-4d7d-9108-31eed0a0f3d9":{"modify":{"timestamp":1715214333315}},"e246ed18-34fa-4f0f-987b-aa8a9f3a0512":{"modify":{"timestamp":1715214333315}},"6f2007cd-f719-456e-8b84-b5f2e60a2315":{"modify":{"timestamp":1715214333315}},"79ab3c90-e35e-4ea2-9e4d-04614e2940d4":{"modify":{"timestamp":1715214333315}},"6891af61-2961-46cd-88f1-074a6160a526":{"modify":{"timestamp":17152143333
> 15}},"976b4a68-3624-449b-805f-12f6d7613e9f":{"modify":{"timestamp":1715214333315}},"de7527fb-b330-4e8d-8421-ffe4a2df8795":{"modify":{"timestamp":1715214333315}},"e1a4d92c-1744-4691-93d5-4e1f00a57575":{"modify":{"timestamp":1715214333315}},"99d0028f-d5e8-4863-9c77-c35bfa20abb3":{"modify":{"timestamp":1715214333315}},"6d081ce0-f9bd-4dec-836f-36a8b981f2b9":{"modify":{"timestamp":1715214333315}},"70f9c29d-25b8-4c50-9bdf-7f54bbf9b965":{"modify":{"timestamp":1715214333315}},"3e897aac-3963-49e0-a412-cdeaf3816b91":{"modify":{"timestamp":1715214333315}},"20d4a9c4-e98d-4376-8a55-8932be1f508f":{"modify":{"timestamp":1715214333315}},"8ea51c7d-26c5-419f-9330-b98264a3d004":{"modify":{"timestamp":1715214333315}},"a5ebb2b1-59a6-44b3-beb2-6b77a1b1dcf9":{"modify":{"timestamp":1715214333315}},"2
> 758563c-c060-4914-b1f8-ebb4b8590e96":{"modify":{"timestamp":1715214333315}},"d0b984e5-dc78-4d87-a2d9-2dd3d86e735e":{"modify":{"timestamp":1715214333315}},"ae058441-4ee3-4e98-b8e9-5b9c475b9e46":{"modify":{"timestamp":1715214333315}},"4e1f7705-2343-4805-9030-673a24c59728":{"modify":{"timestamp":1715214333315}},"10e7a830-edff-4d51-9476-19890e8eb4fd":{"modify":{"timestamp":1715214333315}},"00eace99-797f-4585-8cfa-c22b56dfc909":{"modify":{"timestamp":1715214333315}},"fe2b0889-2380-4709-9b95-99b904cbe597":{"modify":{"timestamp":1715214333315}},"cfe89d09-8c25-4ad9-be08-e98e02d4580c":{"modify":{"timestamp":1715214333315}},"e0a18a4d-8309-44a5-9806-bd08fcd128d5":{"modify":{"timestamp":1715214333315}},"baa2e682-df84-41e3-9064-64e60877e991":{"modify":{"timestamp":1715214333315}},"f6b6a221
> -34b9-4d64-854e-668d5e038d98":{"modify":{"timestamp":1715214333315}},"fb862fd6-4e2c-4545-9eec-57041fb37b33":{"modify":{"timestamp":1715214333315}},"c5d251e5-e53e-47b5-8dca-5a9c383037c7":{"modify":{"timestamp":1715214333315}},"2edebfe7-1975-4772-ab75-e39ad9459d1c":{"modify":{"timestamp":1715214333315}},"f005bbdc-4e0e-4338-8a8c-67ce717e428b":{"modify":{"timestamp":1715214333315}},"de56f881-b0dc-400d-81b0-9a33836a40c0":{"modify":{"timestamp":1715214333315}},"3a1bd08c-da92-476e-98cc-4b5df683c69d":{"modify":{"timestamp":1715214333315}},"204ee42f-9e93-4acf-ad0e-03d80b4fa683":{"modify":{"timestamp":1715214333315}},"c0244a46-e625-4706-b9bc-aea2de0a0ead":{"modify":{"timestamp":1715214333315}},"6a5fee1e-8121-4f37-b54a-490670b75858":{"modify":{"timestamp":1715214333315}},"97df5926-2801-4
> e01-9c67-26b9622d9401":{"modify":{"timestamp":1715214333315}},"bc41881f-659f-4f39-8190-b06766b66d11":{"modify":{"timestamp":1715214333315}},"f806e27f-4705-4f15-9add-24c54ea312c7":{"modify":{"timestamp":1715214333315}},"200bf7c2-615d-4020-944f-5f04ec6a9ece":{"modify":{"timestamp":1715214333315}},"2d31f636-1ec1-45ac-bd04-45351841a046":{"modify":{"timestamp":1715214333315}},"0dde8708-d7e2-452b-9d58-7ed4c63f12a2":{"modify":{"timestamp":1715214333315}},"cdd3a293-cb11-4216-83b0-4ebb9b910c6b":{"modify":{"timestamp":1715214333315}},"55689987-6b55-44f3-8600-e4af0c74e718":{"modify":{"timestamp":1715214333315}},"2972e948-42fc-48f9-af53-f05367a78487":{"modify":{"timestamp":1715214333315}},"c3275eb7-e03b-4d1b-b154-761cddf38164":{"modify":{"timestamp":1715214333315}},"568ff817-fc39-424d-8c7
> 4-77413976612f":{"modify":{"timestamp":1715214333315}},"df2742d3-393f-4bc8-bafb-b6ca3bfe9c7d":{"modify":{"timestamp":1715214333315}},"5bd63d77-7fb6-4b3c-b75f-1511c87f2dcc":{"modify":{"timestamp":1715214333315}},"bdd0d86b-1bf7-4d5b-b6c3-d84c4564a534":{"modify":{"timestamp":1715214333315}},"7197169f-cdaa-4572-bd3d-c2af688c4e50":{"modify":{"timestamp":1715214333315}},"0ee283ea-a258-4fee-a390-b528e1be7b4d":{"modify":{"timestamp":1715214333315}},"399a8038-647d-4b42-9239-83c2a4ee5ed8":{"modify":{"timestamp":1715214333315}},"affb029c-4f0b-4296-98f2-37904f3d57d7":{"modify":{"timestamp":1715214333315}},"7ce0e859-b873-44ac-bf8e-6c3fc503e148":{"modify":{"timestamp":1715214333315}},"84fead26-e638-4299-9ffe-506b1ab7d363":{"modify":{"timestamp":1715214333315}},"949329fa-7e0f-4ad1-b200-9906b
> f532a04":{"modify":{"timestamp":1715214333315}},"54669b2f-c1c1-434e-b9ec-6c2964b202e2":{"modify":{"timestamp":1715214333315}},"73a9ba61-2031-4dfb-8c3f-ab1a0da491a2":{"modify":{"timestamp":1715214333315}},"d1fdbd2d-260b-488f-93d1-ba055d347853":{"modify":{"timestamp":1715214333315}},"32ed347a-08fa-42bc-93c5-57c5dcc2fa20":{"modify":{"timestamp":1715214333315}},"18dddc59-c413-4aa5-8124-178ba2b9b7eb":{"modify":{"timestamp":1715214333315}},"369abb8e-adf6-4254-a658-a2b565469452":{"modify":{"timestamp":1715214333315}},"915638e2-4dd4-44f4-8d85-121aeaf85d33":{"modify":{"timestamp":1715214333315}},"ab849240-3f31-424a-bdd7-835022601c4f":{"modify":{"timestamp":1715214333315}},"696d1036-dfde-4dc3-aae7-f5ebbbac6178":{"modify":{"timestamp":1715214333315}},"7bdd95a3-88c3-449c-b6d4-81e8607b2e3d
> ":{"modify":{"timestamp":1715214333315}},"fb772856-b8ec-4206-a14a-e4063ca3a795":{"modify":{"timestamp":1715214333315}},"3bb14009-fe17-4a41-855e-2705961cb4b8":{"modify":{"timestamp":1715214333315}},"cf0b120d-7c99-4324-99b9-7089057823b6":{"modify":{"timestamp":1715214333315}},"27e0a03d-f2e6-4eac-8358-2427a063793a":{"modify":{"timestamp":1715214333315}},"d6f7da43-de4c-49a1-8ee5-bd072d316a7f":{"modify":{"timestamp":1715214333315}},"c9ad2039-2916-4090-91b1-4fbe57c76185":{"modify":{"timestamp":1715214333315}},"83d9fa21-2978-407d-92d9-0c8550284164":{"modify":{"timestamp":1715214333315}},"f7a6170e-9f0e-4e78-8999-044f285c5fb1":{"modify":{"timestamp":1715214333315}},"cbe04706-a1d3-4726-b626-8fec0143a55c":{"modify":{"timestamp":1715214333315}},"831cbce4-310d-4f6e-ac1e-71830613772b":{"mod
> ify":{"timestamp":1715214333315}},"5f5f6060-7700-4a50-8516-de248ea2ec89":{"modify":{"timestamp":1715214333315}},"8d1a553a-6596-4c5a-981c-186cb04763e9":{"modify":{"timestamp":1715214333315}},"ccfd4423-f764-4d91-a641-b4f0371832a4":{"modify":{"timestamp":1715214333315}},"d440d898-d3b4-43a6-b2e3-04532a4f54a3":{"modify":{"timestamp":1715214333315}},"2afa8de2-1330-4fce-9a96-db636cdf466a":{"modify":{"timestamp":1715214333315}},"8938fb78-e727-4011-a9ab-18a4c5c022aa":{"modify":{"timestamp":1715214333315}},"733b37b2-7610-4c86-a72d-c7d219872169":{"modify":{"timestamp":1715214333315}},"56925ffb-490b-4e52-bdc1-20dc6a1b6dd5":{"modify":{"timestamp":1715214333315}},"b1bb94d3-11c0-48af-a794-695d6b3fb788":{"modify":{"timestamp":1715214333315}},"df5984b7-221f-41aa-8721-1fa892b22666":{"modify":{"
> timestamp":1715214333315}},"3c11b164-fa9d-4170-8d25-414f3a238bd7":{"modify":{"timestamp":1715214333315}},"31580e56-7e66-4047-b95a-3d080a66b5e6":{"modify":{"timestamp":1715214333315}},"202c1d7b-2c86-4f9f-af37-4b8cc9399552":{"modify":{"timestamp":1715214333315}},"7461a718-0f69-4a13-8821-23fbdc3cfcbf":{"modify":{"timestamp":1715214333315}},"169c8666-10e6-45eb-9d17-33a66b5fc98f":{"modify":{"timestamp":1715214333315}},"484eef76-df0c-4fb6-9ef2-79273047d188":{"modify":{"timestamp":1715214333315}},"780d5c54-542b-4ef6-9b4b-8c7af1230a3d":{"modify":{"timestamp":1715214333315}},"5d0b2eb6-c686-454f-a762-b3993e3cb4e3":{"modify":{"timestamp":1715214333315}},"cad7e8d9-93a9-4378-b884-e7d4d6cec6e9":{"modify":{"timestamp":1715214333315}},"22b780d7-3d3e-45cb-938b-4864ccf3834f":{"modify":{"timesta
> mp":1715214333315}},"8b8a5f5d-543b-4209-a739-39d4cf3fa7d5":{"modify":{"timestamp":1715214333315}},"23992016-2b75-4dd2-8a24-c980b94658ac":{"modify":{"timestamp":1715214333315}},"4f7869a6-e31e-4b9e-9724-8297c8580caa":{"modify":{"timestamp":1715214333315}},"2391a35d-762d-4851-8fd1-0b18362f287d":{"modify":{"timestamp":1715214333315}},"264178f2-38b3-4706-bea2-dc8ceeec21a2":{"modify":{"timestamp":1715214333315}},"28a143ad-dff8-4b17-a6a9-db3370da0b64":{"modify":{"timestamp":1715214333315}},"68efd92e-1e78-4976-97e0-e7475134db34":{"modify":{"timestamp":1715214333315}},"17c87180-accd-47e4-b449-fa2e4c3b02ae":{"modify":{"timestamp":1715214333315}},"c9767f21-6c7c-4f4c-a3b1-c5e53fad901f":{"modify":{"timestamp":1715214333315}},"d4a7e30b-fe6b-4dad-9ba8-58332025548e":{"modify":{"timestamp":171
> 5214333315}},"444fdb38-567f-4e15-9f89-57d38e048505":{"modify":{"timestamp":1715214333315}},"207856cb-b8fc-4a1d-a452-9c12868fee01":{"modify":{"timestamp":1715214333315}},"791f3333-440d-497e-838e-f142443c8a74":{"modify":{"timestamp":1715214333315}},"6be4e98c-5618-435d-9727-f144892f869c":{"modify":{"timestamp":1715214333315}},"d89c4f89-46ce-42cc-82e8-7c43b5e6717c":{"modify":{"timestamp":1715214333315}},"c390bf93-b8fe-4eca-97f0-c2a87d730a9d":{"modify":{"timestamp":1715214333315}},"86c177b5-4a01-4559-bcba-f226d5fbba2f":{"modify":{"timestamp":1715214333315}},"b31b970a-3809-43b9-85a2-e8b6d08fd6d7":{"modify":{"timestamp":1715214333315}},"81be6dbe-7947-4c28-b4bf-5fed6af82046":{"modify":{"timestamp":1715214333315}},"3e210a85-fbe7-4b45-8c79-c8a5e7c958a2":{"modify":{"timestamp":1715214333
> 315}},"5d1769ef-caa8-40c7-bafe-7164f64f60c9":{"modify":{"timestamp":1715214333315}},"d00f59c0-502e-47ae-9de8-36ac6db41c8e":{"modify":{"timestamp":1715214333315}},"5c49b1dc-c8ac-4dd0-b7ef-11257f40ab26":{"modify":{"timestamp":1715214333315}},"2fbb2e52-8526-4c46-8e55-eee9b827bbdd":{"modify":{"timestamp":1715214333315}},"8d5db582-ae38-478f-8ef5-21c820c72226":{"modify":{"timestamp":1715214333315}},"63719a1b-6d24-4d1f-8218-cc67977dd806":{"modify":{"timestamp":1715214333315}},"6e77088b-174c-402b-ba16-e90eefb0c332":{"modify":{"timestamp":1715214333315}},"2de74d49-1c44-4ff8-9ad1-141d2f1fc2f1":{"modify":{"timestamp":1715214333315}},"4528479e-0467-40e6-bd43-109403df5a41":{"modify":{"timestamp":1715214333315}},"ecf64061-8421-4e8b-82eb-e3bd8546ff71":{"modify":{"timestamp":1715214333315}},"
> a170e439-09a5-47bc-8cac-0cba918c950d":{"modify":{"timestamp":1715214333315}},"05fb3fed-10a8-4ece-90e1-6c17f217485b":{"modify":{"timestamp":1715214333315}},"34fb1b3c-f94b-4650-aee6-7caa691c9254":{"modify":{"timestamp":1715214333315}},"8b481a4c-121e-493c-9917-f0cc234cff7d":{"modify":{"timestamp":1715214333315}},"8fefce5b-9721-467d-b80e-2789fb13a725":{"modify":{"timestamp":1715214333315}},"e7b0659e-908f-4e88-8e59-f6e9eaef2b2b":{"modify":{"timestamp":1715214333315}},"e4bfb889-fba5-48af-bf17-0ed2c8d0280e":{"modify":{"timestamp":1715214333315}},"98e20398-35bd-4a39-8b56-098d3cc960c5":{"modify":{"timestamp":1715214333315}},"4ed7eb3f-6290-4722-89f8-0d316985c707":{"modify":{"timestamp":1715214333315}},"92d1e5ad-7354-4126-abf1-2b9261b6250b":{"modify":{"timestamp":1715214333315}},"61270e1
> f-bf2d-414a-a618-a934b943073b":{"modify":{"timestamp":1715214333315}},"1a3abec8-454b-41df-ba3c-e9bb0ca3a329":{"modify":{"timestamp":1715214333315}},"aec4c8d9-e766-488f-b092-216e5ee9871e":{"modify":{"timestamp":1715214333315}},"a34388f2-fd78-4af4-8a99-0e7789bb9657":{"modify":{"timestamp":1715214333315}},"e665bd3d-5755-49df-8fad-18f48a5a35c1":{"modify":{"timestamp":1715214333315}},"4f15f734-da0f-4015-8f86-4b986195a4a9":{"modify":{"timestamp":1715214333315}},"4e0a6681-2c68-45b5-a469-77b2e2c28ad1":{"modify":{"timestamp":1715214333315}},"1e3382f8-9dae-4615-b924-424a02e8a429":{"modify":{"timestamp":1715214333315}},"c240aab9-6ee2-4440-b669-678d6fa8096e":{"modify":{"timestamp":1715214333315}},"fd01ccc3-3097-4ed3-a944-6a02487a4c57":{"modify":{"timestamp":1715214333315}},"cc2d4975-b8b1-
> 4b51-a98e-b203b997eb16":{"modify":{"timestamp":1715214333315}},"5440a98e-ea6b-4bbb-8b2c-8696f0b88cba":{"modify":{"timestamp":1715214333315}},"2ecedf5b-d7a5-4c44-8a33-159b45e99545":{"modify":{"timestamp":1715214333315}},"2be651fd-0c97-49f1-8528-5b1aa6b8100f":{"modify":{"timestamp":1715214333315}},"327cc9b3-24ef-4edc-b089-f54a6aec973d":{"modify":{"timestamp":1715214333315}},"0fb71f82-e4f5-4a2c-a816-ff6ff874d3f3":{"modify":{"timestamp":1715214333315}},"5079d69d-a791-4ef2-8628-9d9284c162c0":{"modify":{"timestamp":1715214333315}},"334b670f-ebfc-4cf2-a621-c75e5c06ca22":{"modify":{"timestamp":1715214333315}},"162c494d-18d1-4b21-a696-44e4fbc62715":{"modify":{"timestamp":1715214333315}},"cf944de2-9956-46a3-ad06-089cf6c68d8d":{"modify":{"timestamp":1715214333315}},"bc1765d6-0c04-4a4c-82
> 95-6c4bad02d119":{"modify":{"timestamp":1715214333315}},"26b6780d-6705-4d25-8cdc-07d0736ef043":{"modify":{"timestamp":1715214333315}},"6ef760b8-18fb-435a-9baa-6294baa90285":{"modify":{"timestamp":1715214333315}},"d750b51f-5794-4371-8357-a237e69c59bc":{"modify":{"timestamp":1715214333315}},"435cae6d-ae89-422d-9519-374755118d26":{"modify":{"timestamp":1715214333315}},"e71f685f-1389-4474-aa82-ee011ea5aa2d":{"modify":{"timestamp":1715214333315}},"37b4b359-07cd-4d48-a1a4-13befa91f9fd":{"modify":{"timestamp":1715214333315}},"18acbf35-1ef1-4fd8-93ee-34afdba771ad":{"modify":{"timestamp":1715214333315}},"a85f80a2-7dbf-4187-a655-51e099569ce6":{"modify":{"timestamp":1715214333315}},"e7f52319-acd7-456e-a74d-259a753da344":{"modify":{"timestamp":1715214333315}},"76e4a870-4394-433c-8287-4249
> 939f9808":{"modify":{"timestamp":1715214333315}},"d1dbc4e1-f821-43cc-8b69-7776b07b7cbb":{"modify":{"timestamp":1715214333315}},"f22bccfd-4b72-42dd-ba0d-9e4628bfc1de":{"modify":{"timestamp":1715214333315}},"f270bfb6-b873-4101-8824-d7b41a4901db":{"modify":{"timestamp":1715214333315}},"5cf9c793-3bdd-4299-a82f-7cb2621dd4e1":{"modify":{"timestamp":1715214333315}},"7079d04e-dcbd-458b-89b7-bdccc5ad9f42":{"modify":{"timestamp":1715214333315}},"507c8331-1d22-47f0-bacc-9bf958416b72":{"modify":{"timestamp":1715214333315}},"71086d51-4cb6-4b6d-8ac5-acf21ac15189":{"modify":{"timestamp":1715214333315}},"840319a0-374a-4798-998f-6e48bf506c80":{"modify":{"timestamp":1715214333315}},"3acf375f-317e-43d2-bcfa-b107e237f391":{"modify":{"timestamp":1715214333315}},"b5e2dd6c-488b-4bd7-8ac6-2a3770ee2f6
> 9":{"modify":{"timestamp":1715214333315}},"52edf9e3-4869-4212-b1aa-fa143749a8d2":{"modify":{"timestamp":1715214333315}},"cafb71d0-dcf5-4f6e-bbea-8292792c8d21":{"modify":{"timestamp":1715214333315}},"c55afda7-88ba-4b89-940d-8c494a709c3d":{"modify":{"timestamp":1715214333315}},"0d8ba07f-ae5c-43a9-91f3-5f6722c68ee2":{"modify":{"timestamp":1715214333315}},"4a90dcdc-1bb6-4a6f-8da9-e3fc4c960420":{"modify":{"timestamp":1715214333315}},"7ae32f21-178d-462c-b623-63699d7a97fc":{"modify":{"timestamp":1715214333315}},"760a0a37-2080-4538-8b57-4cbf00883d53":{"modify":{"timestamp":1715214333315}},"d601c2eb-54d5-46f8-9f81-2210a9862889":{"modify":{"timestamp":1715214333315}},"d8e108ec-76ab-4ce4-bf0d-6cb1717b222d":{"modify":{"timestamp":1715214333315}},"a033598a-55ce-4088-a192-4b0e4cbe89b8":{"mo
> dify":{"timestamp":1715214333315}},"def7501e-156e-442e-a3d2-6837b004b931":{"modify":{"timestamp":1715214333315}},"6899fee7-6598-4d23-8e1a-09a6c954053d":{"modify":{"timestamp":1715214333315}},"5d5227f6-aed9-4a1f-8be8-655d0e4187ea":{"modify":{"timestamp":1715214333315}},"8cf28b98-c320-4948-b3bb-647d07d60b0f":{"modify":{"timestamp":1715214333315}},"fcd5b445-fdf1-45b9-bd10-8541e0e2eb25":{"modify":{"timestamp":1715214333315}},"e4d2a8c9-12f0-4acb-97cf-6a39eac0877b":{"modify":{"timestamp":1715214333315}},"b5d92068-76f6-4065-9d7a-882e893716e6":{"modify":{"timestamp":1715214333315}},"02261d27-ad9e-464b-b467-eb64f3af0b36":{"modify":{"timestamp":1715214333315}},"342511bb-7f03-49ec-8ac5-ff4c7b7d3cac":{"modify":{"timestamp":1715214333315}},"11e9c9f2-9a40-4f25-8bef-e72818cdcc5a":{"modify":{
> "timestamp":1715214333315}},"cbfc0493-ef57-4650-8b2f-c30e59272d63":{"modify":{"timestamp":1715214333315}},"ababc329-301d-45ee-baec-c8ee179433d1":{"modify":{"timestamp":1715214333315}},"971461e9-7f59-4384-8d7f-b14b6d67eb32":{"modify":{"timestamp":1715214333315}},"37c0e0b0-41ea-40dd-aa67-a58daee871e7":{"modify":{"timestamp":1715214333315}},"3da23864-7369-47d0-be2f-1b9913a4b0ed":{"modify":{"timestamp":1715214333315}},"aa4ea278-2063-437a-9c1a-f34db5cf9aaf":{"modify":{"timestamp":1715214333315}},"9578e13c-a4be-4dd1-a3d2-924da145534b":{"modify":{"timestamp":1715214333315}},"5b1cc626-4345-40ec-a5c1-4e24b4e5c17d":{"modify":{"timestamp":1715214333315}},"7e99b00a-725e-4840-9dc7-db47ce70a3e5":{"modify":{"timestamp":1715214333315}},"62ed49c5-3ff8-4a22-beac-74d49a27fdc8":{"modify":{"timest
> amp":1715214333315}},"12b08cdc-ddfb-4231-8a16-09cbdc37b0fc":{"modify":{"timestamp":1715214333315}},"eaa4e421-8aa3-43f4-a3f5-bc9a3146be7b":{"modify":{"timestamp":1715214333315}},"99d5e073-49e8-4dd7-9a51-c69c7fccfe2e":{"modify":{"timestamp":1715214333315}},"16e77aaf-0702-4bf3-95c6-0e7fd81348b7":{"modify":{"timestamp":1715214333315}},"23442d68-94bf-4253-9049-cb3c8ac7f155":{"modify":{"timestamp":1715214333315}},"86a62db8-59f6-4459-bcd0-dda0d04e78c5":{"modify":{"timestamp":1715214333315}},"cc8bf87c-e129-4044-ae64-ee5c574fbc46":{"modify":{"timestamp":1715214333315}},"be5ee1d7-070d-4b7f-9769-db9018fba805":{"modify":{"timestamp":1715214333315}},"fbdf6205-1333-4a62-9cdf-3e087e226ee1":{"modify":{"timestamp":1715214333315}},"07385b41-bee6-4308-82f5-63e61ccdb33c":{"modify":{"timestamp":17
> 15214333315}},"29cac1e4-4e3a-428d-93ae-65cea3172c31":{"modify":{"timestamp":1715214333315}},"f4d264a8-487f-439b-8827-8bbe49e492dd":{"modify":{"timestamp":1715214333315}},"6a8002f8-dd26-40f1-a380-81acf229cbb4":{"modify":{"timestamp":1715214333315}},"3dbbd2a9-ece9-4f0f-86bd-e612aa5ab675":{"modify":{"timestamp":1715214333315}},"84bc9478-f4a5-4000-ac5e-cf98f4c8c35a":{"modify":{"timestamp":1715214333315}},"742f002b-9508-4d6c-a2a1-ea41511fc69d":{"modify":{"timestamp":1715214333315}},"67f46fb6-3a11-4fe8-9107-f46765361609":{"modify":{"timestamp":1715214333315}},"0aa1495e-61ec-45c6-a67b-23ae0dc9014f":{"modify":{"timestamp":1715214333315}},"95523c36-0888-42a6-9eeb-2d4b328eb54c":{"modify":{"timestamp":1715214333315}},"d26eea21-fa2e-479c-bd6b-215b350675b5":{"modify":{"timestamp":171521433
> 3315}},"e0fca8aa-2b70-4037-aa14-716353741908":{"modify":{"timestamp":1715214333315}},"e741c687-ae87-497d-a712-348735afa259":{"modify":{"timestamp":1715214333315}},"f07a1380-14e3-4117-807c-90462d829fff":{"modify":{"timestamp":1715214333315}},"a92b279d-cf45-41ba-8665-6237567e3c1c":{"modify":{"timestamp":1715214333315}},"9083e08a-4e7a-4f28-adcf-5be45dd46ae8":{"modify":{"timestamp":1715214333315}},"977efb31-ccc3-4ad5-988a-f65204f81ed5":{"modify":{"timestamp":1715214333315}},"b3c3d648-7ef9-41c2-8664-d22c9456f0c0":{"modify":{"timestamp":1715214333315}},"f120d177-6769-49d9-818e-dde147b0da72":{"modify":{"timestamp":1715214333315}},"7289d1e7-ce79-4b68-82a3-014a55ae03e3":{"modify":{"timestamp":1715214333315}},"4bc64441-b149-418a-b4a8-e3b948ddc3b3":{"modify":{"timestamp":1715214333315}},
> "1726c6aa-9ba2-41c2-b40c-f8dfa49c357e":{"modify":{"timestamp":1715214333315}},"b57dc0d8-3cfd-4bec-955d-71c97a81159f":{"modify":{"timestamp":1715214333315}},"8de57e71-9b90-4de3-94ba-64c73673def2":{"modify":{"timestamp":1715214333315}},"1b3cac63-26db-4afb-9d54-047f2cd0e160":{"modify":{"timestamp":1715214333315}},"dbc56d3f-6cf0-4e99-af38-13bfdb5aa65e":{"modify":{"timestamp":1715214333315}},"3a005913-a304-47ae-8f7b-143f0d6e0f04":{"modify":{"timestamp":1715214333315}},"078ea5c9-c415-474b-a980-4f4ea419f7e5":{"modify":{"timestamp":1715214333315}},"08711c85-fbe1-48ea-98f9-738deb4604b6":{"modify":{"timestamp":1715214333315}},"07c9f836-255d-4821-9437-a07595dce6d9":{"modify":{"timestamp":1715214333315}},"60caf149-d38d-4d3a-a351-ed4876f14ff2":{"modify":{"timestamp":1715214333315}},"49752a
> ea-2f9e-411e-a2aa-1e3c962e3397":{"modify":{"timestamp":1715214333315}},"473f058b-320e-4451-a0c3-4a84f6c89d16":{"modify":{"timestamp":1715214333315}},"cae869af-c4bf-4d91-81d2-b6addd514c9d":{"modify":{"timestamp":1715214333315}},"18c3c0a2-9331-484a-8f55-e72253c2b9a2":{"modify":{"timestamp":1715214333315}},"0910e345-94a4-4a08-b350-292b2a8dca96":{"modify":{"timestamp":1715214333315}},"c4b171d5-572c-4b4e-9f28-b94dbf7154d7":{"modify":{"timestamp":1715214333315}},"203398e2-dd12-4f2a-9aa5-30b893c3c8b0":{"modify":{"timestamp":1715214333315}},"ac6a8156-7f00-4590-9979-becb94d1bd80":{"modify":{"timestamp":1715214333315}},"005023bb-9bf6-4aeb-b44c-471471cede03":{"modify":{"timestamp":1715214333315}},"162ef89a-c2a9-47a9-b13a-533e989c6170":{"modify":{"timestamp":1715214333315}},"8b9927d9-2f7f
> -4e52-8497-083da3447dd6":{"modify":{"timestamp":1715214333315}},"b9a6eaa3-9b05-46de-a345-6f1b4180e7ad":{"modify":{"timestamp":1715214333315}},"652d60c2-34e1-45e0-b697-dfd560f44925":{"modify":{"timestamp":1715214333315}},"6bcb6234-724d-4bf3-985b-01aa1d5c4ea1":{"modify":{"timestamp":1715214333315}},"6c832846-f50f-4054-98ba-b2a25c134729":{"modify":{"timestamp":1715214333315}},"675b7ed5-b460-4f39-b60d-96d6530d1ad4":{"modify":{"timestamp":1715214333315}},"483a3493-e2f6-4947-91c4-0ec0836866da":{"modify":{"timestamp":1715214333315}},"3602eb2c-0b67-4bfc-85e7-0e351c71661b":{"modify":{"timestamp":1715214333315}},"9e31a26a-2a9a-416b-97fc-cc19098bf085":{"modify":{"timestamp":1715214333315}},"5dab9a2e-84da-400f-9dad-2c50577b1bc3":{"modify":{"timestamp":1715214333315}},"1b7fa43d-ded3-4c71-b
> 9b7-1268d79e9555":{"modify":{"timestamp":1715214333315}},"9af0fe88-3117-45b7-b001-2a88f6a5280e":{"modify":{"timestamp":1715214333315}},"5ba8bdad-ae63-496b-9e7e-6dc822d9ecfb":{"modify":{"timestamp":1715214333315}},"f9db402b-001d-4f44-af86-fb308ec63a49":{"modify":{"timestamp":1715214333315}},"49957476-0033-449e-a658-dee8f3ef9246":{"modify":{"timestamp":1715214333315}},"ebb58924-e346-4ed7-8892-912afb956b6e":{"modify":{"timestamp":1715214333315}},"acb1ec32-924d-42ae-b158-d6f5ff026ef9":{"modify":{"timestamp":1715214333315}},"aca366ea-42fa-4a92-b825-2deba8b39b0c":{"modify":{"timestamp":1715214333315}},"6fa6d314-3916-409c-8989-1bb60c8fb05f":{"modify":{"timestamp":1715214333315}},"9be623ef-9fa3-4955-a53b-6cb546ec9f0b":{"modify":{"timestamp":1715214333315}},"17ff3e7c-b4be-49c4-a57f-b18
> bc8c5482a":{"modify":{"timestamp":1715214333315}},"65a1e74c-8c99-4711-944d-1f0ae9bf0aed":{"modify":{"timestamp":1715214333315}},"5838f0f3-7efa-450d-93a7-e427ebce1357":{"modify":{"timestamp":1715214333315}},"45833265-adc6-4b89-b18b-407308c7a3c0":{"modify":{"timestamp":1715214333315}},"5c93edb7-6a61-41c8-a1aa-5eaae6af92a8":{"modify":{"timestamp":1715214333315}},"99992606-51dd-4f7c-a042-b638632abbb0":{"modify":{"timestamp":1715214333315}},"69d87a80-8bce-4e6e-9978-3087faa8d5ca":{"modify":{"timestamp":1715214333315}},"e0092dc8-5eca-418e-977f-f3c83a3f039b":{"modify":{"timestamp":1715214333315}},"893c6a74-e21c-4b0a-984b-844da72a924b":{"modify":{"timestamp":1715214333315}},"46d39e3b-4dc5-4e86-b3df-a708827f3092":{"modify":{"timestamp":1715214333315}},"6d54014f-e0d4-438b-b672-303da22bf9
> 85":{"modify":{"timestamp":1715214333315}},"a1da81db-992b-4977-ad78-e386b82ad51e":{"modify":{"timestamp":1715214333315}},"225a991b-822a-4a48-9b60-4bc16b39ca29":{"modify":{"timestamp":1715214333315}},"302fc5ee-6518-4657-83b3-48646e465b0c":{"modify":{"timestamp":1715214333315}},"3d303e41-0c92-4cd4-8e20-e8552cd62241":{"modify":{"timestamp":1715214333315}},"1568509a-0762-4a02-bff0-20885052dd20":{"modify":{"timestamp":1715214333315}},"7454c0e5-7204-4f34-a837-735adf165562":{"modify":{"timestamp":1715214333315}},"b730b0af-8acf-4813-a5b3-558acf3e4274":{"modify":{"timestamp":1715214333315}},"04bb21db-e840-4e39-94d0-59865df4dcaf":{"modify":{"timestamp":1715214333315}},"f5cac95e-d8d8-49e4-ab57-4552a043e6a6":{"modify":{"timestamp":1715214333315}},"eed38a59-9a52-4615-9a18-0a4bc475bc0f":{"m
> odify":{"timestamp":1715214333315}},"b88dd72b-562b-4b08-b7ec-97b6cdb0cbb7":{"modify":{"timestamp":1715214333315}},"8a2b50db-1133-437e-aaad-eabaff7e44cc":{"modify":{"timestamp":1715214333315}},"3b9d2f4e-c487-4c4a-9e78-27302ae03e89":{"modify":{"timestamp":1715214333315}},"1a0e66d4-1943-4b15-a01f-dcbe0a4d6536":{"modify":{"timestamp":1715214333315}},"bce17030-02ad-4682-ab31-4ff0faf36579":{"modify":{"timestamp":1715214333315}},"70a100aa-4454-4ae9-ac14-c413ff576e9f":{"modify":{"timestamp":1715214333315}},"1b681afa-1840-4954-a75b-f93fae537096":{"modify":{"timestamp":1715214333315}},"2a74f32b-7e11-4575-99b4-56d36eab6a30":{"modify":{"timestamp":1715214333315}},"fb01ee23-8ff1-456b-939f-f1edce6ea59e":{"modify":{"timestamp":1715214333315}},"4fd16807-3e4f-4762-805f-8001e799271a":{"modify":
> {"timestamp":1715214333315}},"05bf10db-a393-4f4b-a7e0-9b35b154ad87":{"modify":{"timestamp":1715214333315}},"2c1aeaf7-8137-471b-951e-743fe3d5121a":{"modify":{"timestamp":1715214333315}},"cd79c727-fe6d-485f-9549-71e4ef52f077":{"modify":{"timestamp":1715214333315}},"0fc146ef-fafe-47a1-ae36-5a3902ca3ecb":{"modify":{"timestamp":1715214333315}},"2b7bd047-1f6e-407b-bdd5-5a221e3addd5":{"modify":{"timestamp":1715214333315}},"4033d7d5-5d51-4738-8138-07333ffbecb9":{"modify":{"timestamp":1715214333315}},"a58d57a2-bfda-4b04-a214-07b579d428a8":{"modify":{"timestamp":1715214333315}},"4b67bc9d-6423-4787-8b8e-f0043be95505":{"modify":{"timestamp":1715214333315}},"7cdd03b7-e4b3-4715-a59b-71c516945323":{"modify":{"timestamp":1715214333315}},"dfb95178-3296-428c-94da-e3de7c180d68":{"modify":{"times
> tamp":1715214333315}},"b6c9000d-d381-4de0-aba9-5df788740298":{"modify":{"timestamp":1715214333315}},"8ae4a34f-2a76-47bb-9b78-cc9702a7f7bc":{"modify":{"timestamp":1715214333315}},"151479ee-1bec-4a08-9767-664d137a8e8a":{"modify":{"timestamp":1715214333315}},"86ede7d9-2f6e-4e65-be2a-769b92a8a6c5":{"modify":{"timestamp":1715214333315}},"74623090-3e6f-475b-aa8e-e4b27a90e98a":{"modify":{"timestamp":1715214333315}},"827bfb76-7aa6-4605-bbbd-aca33eee1cf6":{"modify":{"timestamp":1715214333315}},"96f2ee76-f5f9-453d-9566-bec1a27d62c4":{"modify":{"timestamp":1715214333315}},"9a9d7d62-ec6a-431e-b3f7-67e1c72bcd94":{"modify":{"timestamp":1715214333315}},"8398008a-625b-4d96-b934-49365b174dc7":{"modify":{"timestamp":1715214333315}},"40a76fb5-47c5-437e-a77c-6283f14aa56b":{"modify":{"timestamp":1
> 715214333315}},"1ecc90a1-823d-4412-90b0-7c9b4cc06d0b":{"modify":{"timestamp":1715214333315}},"0cd4ae46-619f-4d90-9ef4-29547ef329f0":{"modify":{"timestamp":1715214333315}},"e638ea56-6e97-424d-b1fd-2f9a2de8dab6":{"modify":{"timestamp":1715214333315}},"8d1ee17c-c06a-4f92-88ca-128684710599":{"modify":{"timestamp":1715214333315}},"e35c4f6d-bbe3-47f6-a042-124cc278c680":{"modify":{"timestamp":1715214333315}},"ca971702-e6a5-4d91-8bcb-86e57ff8ad9b":{"modify":{"timestamp":1715214333315}},"61a92c28-e246-4720-a50c-97a07bdb79c8":{"modify":{"timestamp":1715214333315}},"970fec83-3c75-43e6-97ff-3fbfce862fe2":{"modify":{"timestamp":1715214333315}},"a806c381-6685-4572-8430-b248cf809db6":{"modify":{"timestamp":1715214333315}},"a21189cf-af0f-455e-bceb-1f436b6c497e":{"modify":{"timestamp":17152143
> 33315}},"61d33e58-6eba-410d-89ca-10292eebe2a9":{"modify":{"timestamp":1715214333315}},"7a076659-51fd-40ff-bf16-de9e0cda4a5e":{"modify":{"timestamp":1715214333315}},"cfeb6f56-4a7a-40bd-a7ed-e11631988122":{"modify":{"timestamp":1715214333315}},"f73fa86a-6a5e-44f6-8abb-61b3eb1b718e":{"modify":{"timestamp":1715214333315}},"00301735-f3ce-461d-b9a9-7e7b85684501":{"modify":{"timestamp":1715214333315}},"f1f79bd6-c447-4f77-89a8-ed0c05076f7e":{"modify":{"timestamp":1715214333315}},"715094bc-d4af-4fcf-897a-bf3369f7d974":{"modify":{"timestamp":1715214333315}},"a0d0b6e3-5e71-4c9b-8697-ff77662e5af1":{"modify":{"timestamp":1715214333315}},"d8c1eafb-4e6f-4ce8-8a47-c839f476a216":{"modify":{"timestamp":1715214333315}},"920a37b2-2d25-4b8f-9ec7-8fae4a6bd17c":{"modify":{"timestamp":1715214333315}}
> ,"5e1e82df-e290-497d-bed4-c107fe8ac9d0":{"modify":{"timestamp":1715214333315}},"864e22d4-396f-4320-afc7-ce703e14a535":{"modify":{"timestamp":1715214333315}},"3f3de9e9-15ed-48c6-9b1f-1a66f891c783":{"modify":{"timestamp":1715214333315}},"e3b15923-66bc-46a9-b980-f0a8cf4b77a1":{"modify":{"timestamp":1715214333315}},"e86eea29-564a-47d7-aaec-392d870d518e":{"modify":{"timestamp":1715214333315}},"899a9edc-9782-4302-8a6d-fa6dc40506ea":{"modify":{"timestamp":1715214333315}},"9ffa37dc-2677-4b20-a300-2d141c18ab42":{"modify":{"timestamp":1715214333315}},"4794826e-5588-4277-8062-bcfc8f99f33f":{"modify":{"timestamp":1715214333315}},"39769d67-960c-4782-bdcd-903ffe86b5e9":{"modify":{"timestamp":1715214333315}},"a970737b-c22e-4d81-8f55-4ee2336542de":{"modify":{"timestamp":1715214333315}},"c9b43
> 571-7e1e-4994-a9e5-cc516e89d54c":{"modify":{"timestamp":1715214333315}},"941b3942-f346-4fb4-a6d6-52afe163a5ec":{"modify":{"timestamp":1715214333315}},"c4ce4ccc-2045-492b-a71d-07bf65c928a3":{"modify":{"timestamp":1715214333315}},"d4ab4817-1954-4072-8249-1424611be797":{"modify":{"timestamp":1715214333315}},"28adc332-db03-42e4-b72f-d368cd610013":{"modify":{"timestamp":1715214333315}},"0c2dc75a-dc51-4e9c-9882-54b0c708f389":{"modify":{"timestamp":1715214333315}},"5dbfd49b-f78d-41dc-9a7c-30810bc7e531":{"modify":{"timestamp":1715214333315}},"8b3c0b4f-95ce-48b0-8787-6793714bfa54":{"modify":{"timestamp":1715214333315}},"64ca97f2-bb61-4d7e-a032-c46e63883fd5":{"modify":{"timestamp":1715214333315}},"243438f7-13cb-467b-8bef-3883460501dd":{"modify":{"timestamp":1715214333315}},"0fd6ee90-8c5
> c-49b1-9b14-2cb7f5dc8d92":{"modify":{"timestamp":1715214333315}},"87403bcb-2d4c-4cae-bd53-3631c260fdd8":{"modify":{"timestamp":1715214333315}},"659ba3dc-b7ce-4277-8029-b9fc0e246218":{"modify":{"timestamp":1715214333315}},"12ed4176-6623-47a1-980e-daf39a88d1bc":{"modify":{"timestamp":1715214333315}},"9bd2fc72-ee20-495d-8f35-e8060d014341":{"modify":{"timestamp":1715214333315}},"0257ddfb-9af1-408a-a63e-ff0f78728425":{"modify":{"timestamp":1715214333315}},"4455832c-8936-4b89-b8f6-7b669c904c7a":{"modify":{"timestamp":1715214333315}},"fe6cf29c-5c2a-4a1a-8e8f-177246fa7fc4":{"modify":{"timestamp":1715214333315}},"d6350efd-328d-486b-8a00-17ef64577f0a":{"modify":{"timestamp":1715214333315}},"04cdb37a-2ea3-44b2-b246-671d57a29da9":{"modify":{"timestamp":1715214333315}},"b67af17e-709b-401a-
> b95c-dc99fed813c3":{"modify":{"timestamp":1715214333315}},"d5ed481c-97b9-4a6d-bb6f-fcc0eb066918":{"modify":{"timestamp":1715214333315}},"fe8c5678-069e-44c8-b2c6-55350bdcda74":{"modify":{"timestamp":1715214333315}},"66a933db-c7b5-4191-907c-5090f670c56b":{"modify":{"timestamp":1715214333315}},"405ca5e6-2e3d-4c24-98d2-f10e8764bae3":{"modify":{"timestamp":1715214333315}},"9f16fbe8-46b6-477b-b620-fe942c9d50bc":{"modify":{"timestamp":1715214333315}},"21ae72fb-642c-417a-a770-6342db8ec4d1":{"modify":{"timestamp":1715214333315}},"c3394240-8c39-4813-87c0-0ec6a267746c":{"modify":{"timestamp":1715214333315}},"2a7fba28-5ede-4833-b899-419199998df1":{"modify":{"timestamp":1715214333315}},"35ece4b5-7c76-43a8-9e66-2d80ec81f37f":{"modify":{"timestamp":1715214333315}},"62f4289c-4f82-498b-9b09-e9
> 5c474ccea6":{"modify":{"timestamp":1715214333315}},"1e72239e-07f4-4a31-9dcd-59ade29a266c":{"modify":{"timestamp":1715214333315}},"4f2163d8-7497-4d88-8db3-b796931aaba3":{"modify":{"timestamp":1715214333315}},"07943ef1-4b15-4572-9963-f9d5b82d70c7":{"modify":{"timestamp":1715214333315}},"09be50ea-96f0-4ac1-bf7b-755f8fc8c131":{"modify":{"timestamp":1715214333315}},"e9bb66cc-49fc-40a9-82f7-45295fa1dc73":{"modify":{"timestamp":1715214333315}},"e27ddcfd-1ce6-4b1d-b442-ddd5d34aa92b":{"modify":{"timestamp":1715214333315}},"61a7a2f5-e348-442c-919d-d3f13ad97208":{"modify":{"timestamp":1715214333315}},"e03ae449-17d7-4fb8-968b-8fd2c7e6bd65":{"modify":{"timestamp":1715214333315}},"ecf87deb-6b45-4374-8e3b-3f36e068f032":{"modify":{"timestamp":1715214333315}},"9e7708fa-5bc5-4d13-9da8-1b07b5627
> 20c":{"modify":{"timestamp":1715214333315}},"926c8ec2-8b7c-4198-b116-e11759e4467e":{"modify":{"timestamp":1715214333315}},"0248652a-9700-45ce-b892-36863250d0e6":{"modify":{"timestamp":1715214333315}},"b98ed424-cf6b-43b7-a486-a28c927e1714":{"modify":{"timestamp":1715214333315}},"178343c9-c37c-4830-b909-1bd6907b5fa1":{"modify":{"timestamp":1715214333315}},"34708224-927c-4c8a-aa19-6b6d2f46b301":{"modify":{"timestamp":1715214333315}},"c9f5f254-8193-476d-9eae-9e020a7dc82f":{"modify":{"timestamp":1715214333315}},"ead846a1-bbd7-41d8-890a-d90c9ea3836a":{"modify":{"timestamp":1715214333315}},"78351768-2d87-481d-a6db-8f6c5d4883fa":{"modify":{"timestamp":1715214333315}},"aff94ca2-9f41-4e08-a5b0-4e918503048b":{"modify":{"timestamp":1715214333315}},"347331b1-c734-43f7-889d-f7fe57a6d262":{"
> modify":{"timestamp":1715214333315}},"17f681df-1c8a-4536-a30f-ec001083d095":{"modify":{"timestamp":1715214333315}},"23380987-b3d2-4949-b230-5091e7f65ee3":{"modify":{"timestamp":1715214333315}},"2d44fb48-c21e-422a-91bc-76fb881c1f56":{"modify":{"timestamp":1715214333315}},"06154c23-1313-436a-8030-b050249c2369":{"modify":{"timestamp":1715214333315}},"6923900a-7879-4a72-a4a0-6d232af8933b":{"modify":{"timestamp":1715214333315}},"c8cece8c-d5f0-4f63-8ff7-2637c0202d6a":{"modify":{"timestamp":1715214333315}},"fcbe0695-2a0c-4792-9a6f-33a0c1955111":{"modify":{"timestamp":1715214333315}},"b020ca33-7c07-432d-8d78-c25f0db36a0d":{"modify":{"timestamp":1715214333315}},"9e558e0b-b7c5-4e95-a8da-a0d03d2427c3":{"modify":{"timestamp":1715214333315}},"637d8a93-4488-46b4-97fe-aea44b407da6":{"modify"
> :{"timestamp":1715214333315}},"ebe5ac98-8bf7-4a27-ad32-e9944c5db948":{"modify":{"timestamp":1715214333315}},"7b7f47b8-c6bb-415f-9dd0-5026e697164a":{"modify":{"timestamp":1715214333315}},"fb26508d-9d2d-4d6a-ab55-72e5967ca3b3":{"modify":{"timestamp":1715214333315}},"00d08aa4-098d-4298-9f98-4dae9e1b0381":{"modify":{"timestamp":1715214333315}},"19ed8bc7-40c5-4a22-866a-2d36b91e3eaa":{"modify":{"timestamp":1715214333315}},"c164b56a-fbd4-4570-b901-2b0a22200697":{"modify":{"timestamp":1715214333315}},"cf84d907-6ef6-42ac-951d-691e75fc4e78":{"modify":{"timestamp":1715214333315}},"bfe71f63-78f5-408f-9380-e8e445449891":{"modify":{"timestamp":1715214333315}},"6c1f0398-9531-4392-882d-74ed16675dae":{"modify":{"timestamp":1715214333315}},"c10c9d2b-d758-4584-a487-e6b8b42b09b4":{"modify":{"time
> stamp":1715214333315}},"a3357af2-841c-4712-b13b-0ae6e20d87c5":{"modify":{"timestamp":1715214333315}},"b46c47d5-7fed-4d7c-a212-2761bf1edd50":{"modify":{"timestamp":1715214333315}},"cc3e4bda-9446-4543-a874-db59258c1464":{"modify":{"timestamp":1715214333315}},"0b7b588e-6bc8-43dc-970f-6c41b3075d0d":{"modify":{"timestamp":1715214333315}},"6cb9ad85-f3b6-48b3-8541-16575f7b650f":{"modify":{"timestamp":1715214333315}},"7aaea8cd-6555-4bb2-9009-6dddd28607af":{"modify":{"timestamp":1715214333315}},"873a6019-b96e-476b-a328-3644a74da5c4":{"modify":{"timestamp":1715214333315}},"9194901d-b181-4325-a046-4bf86daf0765":{"modify":{"timestamp":1715214333315}},"d996c11f-56a7-4f0e-a9fd-d6224a1fd2b6":{"modify":{"timestamp":1715214333315}},"e9c548f4-7760-4702-aece-75552801954f":{"modify":{"timestamp":
> 1715214333315}},"f4cdb9b8-90d7-41d8-bd1c-a0e67c12da11":{"modify":{"timestamp":1715214333315}},"24c9a233-1762-437c-af53-6b60b44ec9f5":{"modify":{"timestamp":1715214333315}},"97c70a99-d95d-4bb9-9ba5-77cd7ea70902":{"modify":{"timestamp":1715214333315}},"c12a7d6c-f70f-43e5-b0bc-e557ea9030f7":{"modify":{"timestamp":1715214333315}},"5201fc6f-9d10-4757-887e-05419e609c19":{"modify":{"timestamp":1715214333315}},"e0a29478-9370-426e-9618-575515ee1530":{"modify":{"timestamp":1715214333315}},"97b4da23-1b88-4b0f-bb37-69e7e26180b8":{"modify":{"timestamp":1715214333315}},"07da59cd-ddef-48f7-9c50-ccf6bb4793ca":{"modify":{"timestamp":1715214333315}},"a7812fee-dd5f-47b8-bc34-01d65e9c566d":{"modify":{"timestamp":1715214333315}},"f57be720-8a6e-4a62-9a05-8957f7667e03":{"modify":{"timestamp":1715214
> 333315}},"3d2af2de-962e-4ced-9467-8942ab122c98":{"modify":{"timestamp":1715214333315}},"4d08a03d-53d0-435f-8cf1-c9644b8732b9":{"modify":{"timestamp":1715214333315}},"4a6e6864-1c3d-4843-a9aa-a5469208acd0":{"modify":{"timestamp":1715214333315}},"3f90689b-486b-4513-b42f-44ca9548930f":{"modify":{"timestamp":1715214333315}},"b844c4b6-d3bc-4289-9c1e-ed46c8bfe9e4":{"modify":{"timestamp":1715214333315}},"dccd339b-1638-4f36-aaba-fcfe3671f63d":{"modify":{"timestamp":1715214333315}},"23b4d042-c651-4553-a975-dcdbd3d98f25":{"modify":{"timestamp":1715214333315}},"f4975215-5928-483b-8cc6-6462d932e1c1":{"modify":{"timestamp":1715214333315}},"5e2d8fd9-5260-40e8-9f9e-f23870c960a3":{"modify":{"timestamp":1715214333315}},"7ba7fd3d-afe6-42eb-b393-855dbbe21bbe":{"modify":{"timestamp":1715214333315}
> },"5ebe2875-44fa-406b-bd3d-b63c7e9814b1":{"modify":{"timestamp":1715214333315}},"0bb7faa6-a8f7-4643-8d4d-7cec4681d6a8":{"modify":{"timestamp":1715214333315}},"6a016056-81f1-428d-99ca-1bfc9db021fc":{"modify":{"timestamp":1715214333315}},"25df3cd8-4ddc-48b4-95f5-2545438e8a19":{"modify":{"timestamp":1715214333315}},"97fbe8cd-2e19-4380-bcee-ba3fe5374c64":{"modify":{"timestamp":1715214333315}},"902c300f-9993-4a45-9f31-0b1c8cc5c5fc":{"modify":{"timestamp":1715214333315}},"76d751a6-1e63-445e-a6a3-68b0ad3103ab":{"modify":{"timestamp":1715214333315}},"079495b7-70bf-4128-a5a8-c0459f9fdcf3":{"modify":{"timestamp":1715214333315}},"8d5cbeee-5155-4860-9500-09d88998029d":{"modify":{"timestamp":1715214333315}},"b56c8ea4-4188-44ec-aeab-3e33a1dcdcf7":{"modify":{"timestamp":1715214333315}},"e46b
> 6afe-5809-4d82-91ca-0bba39739ee9":{"modify":{"timestamp":1715214333315}},"41949a05-0bb5-4aa0-a47f-45e0f9aaeb75":{"modify":{"timestamp":1715214333315}},"4b6a8f14-7226-4cde-a2e8-00cf53adafdc":{"modify":{"timestamp":1715214333315}},"78042ab7-e747-4792-a9fd-22c898863331":{"modify":{"timestamp":1715214333315}},"55e3838c-89b9-4303-94fe-c33064e6796d":{"modify":{"timestamp":1715214333315}},"3434c77c-b1fb-4969-b4da-5e1ccd3e1d23":{"modify":{"timestamp":1715214333315}}}}]
>
> On Thu, 9 May 2024 at 07:17, Gavin McKee <gavmcke...@googlemail.com> wrote:
> >
> > Numan,
> >
> > We enabled conditional monitoring on the affected machines and still
> > see the issue .
> >
> > It seems crazy that such an important protocol such as DNS is blocked
> > on the main thread. Can someone in the community help with a patch to
> > remove the mutex on the DNS ? How safe will that ultimately be - I'm
> > sure it was there for a reason.
> >
> > I'm not sure where to go from here to be honest.
> >
> > Gav
> >
> > On Wed, 8 May 2024 at 16:51, Numan Siddique <num...@ovn.org> wrote:
> > >
> > > On Wed, May 8, 2024 at 6:01 PM Gavin McKee via discuss
> > > <ovs-discuss@openvswitch.org> wrote:
> > > >
> > > > @Numan Siddique
> > > >
> > > > If we enable conditional monitoring here , will this help ?
> > > > How does transport zones help with something like this ? Do they
> > > > limit the amount of processing . We only have a single VM on this
> > > > node , so single LSP , Logical Switch etc that is actually needed or
> > > > used.
> > > > Would v24.03.0
> > > > https://github.com/ovn-org/ovn/commit/1622526ff2102525e1bbf2ca262842c71d6b9b33
> > > > help here ?
> > > >
> > >
> > > I think conditional monitoring should work as ovn-controllers will not
> > > get updates for changes it's not interested.
> > > You can easily test this out. I'm not sure about the transport zones.
> > > Even if you use transport zones,
> > > conditional monitoring should be enabled, otherwise ovn-controller
> > > will get updates for all SB DB changes.
> > >
> > > I don't think the 24.03 commit you linked to would help in your case
> > > as the issue is with ovn-controller main thread
> > > blocking the pinctrl thread from processing the DNS packet.
> > >
> > > Numan
> > >
> > > > Gav
> > > >
> > > > On Wed, 8 May 2024 at 14:43, Gavin McKee <gavmcke...@googlemail.com>
> > > > wrote:
> > > > >
> > > > > Ok so
> > > > >
> > > > > 1. Customers depend on the internal DNS reccords, so this is needed
> > > > > for production operations
> > > > > 2. I can take a look at the updates - would using conditional
> > > > > monitoring work here? We have ovn-monitor-all=true , would this help
> > > > > at all ?
> > > > > 3 & 4 . Is that something the community can help with? Is that a
> > > > > viable long term fix we could maybe get a patch for ?
> > > > >
> > > > > Gav
> > > > >
> > > > > On Wed, 8 May 2024 at 14:30, Numan Siddique <num...@ovn.org> wrote:
> > > > > >
> > > > > > On Wed, May 8, 2024 at 3:20 PM Gavin McKee via discuss
> > > > > > <ovs-discuss@openvswitch.org> wrote:
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > Can someone help me understand why this issue occurs
> > > > > > >
> > > > > > >
> > > > > > > ovn-controller 23.09.1
> > > > > > > Open vSwitch Library 3.2.2
> > > > > > >
> > > > > > > We have an issue with some machines intermittently unable to
> > > > > > > resolve
> > > > > > > DNS for external domains (example dig +noall +answer
> > > > > > > harmonic-openai-canada.openai.azure.com)
> > > > > > >
> > > > > > > In the OVN controller log I see the following
> > > > > > >
> > > > > > > 2024-05-08T12:12:35.596Z|30138|inc_proc_eng|INFO|node:
> > > > > > > logical_flow_output, recompute (missing handler for input SB_dns)
> > > > > > > took
> > > > > > > 7950ms
> > > > > > > 2024-05-08T14:50:29.747Z|30312|inc_proc_eng|INFO|node:
> > > > > > > logical_flow_output, recompute (missing handler for input SB_dns)
> > > > > > > took
> > > > > > > 8634ms
> > > > > > > 2024-05-08T14:50:46.673Z|30329|inc_proc_eng|INFO|node:
> > > > > > > logical_flow_output, recompute (missing handler for input SB_dns)
> > > > > > > took
> > > > > > > 8774ms
> > > > > > > 2024-05-08T14:54:40.781Z|30353|inc_proc_eng|INFO|node:
> > > > > > > logical_flow_output, recompute (missing handler for input SB_dns)
> > > > > > > took
> > > > > > > 8535ms
> > > > > > > 2024-05-08T14:58:43.381Z|30433|inc_proc_eng|INFO|node:
> > > > > > > logical_flow_output, recompute (missing handler for input SB_dns)
> > > > > > > took
> > > > > > > 8541ms
> > > > > > > 2024-05-08T14:58:56.802Z|30488|inc_proc_eng|INFO|node:
> > > > > > > logical_flow_output, recompute (missing handler for input SB_dns)
> > > > > > > took
> > > > > > > 8820ms
> > > > > > > 2024-05-08T15:02:50.704Z|30512|inc_proc_eng|INFO|node:
> > > > > > > logical_flow_output, recompute (missing handler for input SB_dns)
> > > > > > > took
> > > > > > > 8739ms
> > > > > > > 2024-05-08T15:03:05.206Z|30529|inc_proc_eng|INFO|node:
> > > > > > > logical_flow_output, recompute (missing handler for input SB_dns)
> > > > > > > took
> > > > > > > 8686ms
> > > > > > > 2024-05-08T15:08:39.441Z|30569|inc_proc_eng|INFO|node:
> > > > > > > logical_flow_output, recompute (missing handler for input SB_dns)
> > > > > > > took
> > > > > > > 9167ms
> > > > > > > 2024-05-08T15:09:09.152Z|30603|inc_proc_eng|INFO|node:
> > > > > > > logical_flow_output, recompute (missing handler for input SB_dns)
> > > > > > > took
> > > > > > > 8985ms
> > > > > > > 2024-05-08T15:12:14.361Z|30632|inc_proc_eng|INFO|node:
> > > > > > > logical_flow_output, recompute (missing handler for input SB_dns)
> > > > > > > took
> > > > > > > 8569ms
> > > > > > > 2024-05-08T15:13:52.535Z|30705|inc_proc_eng|INFO|node:
> > > > > > > logical_flow_output, recompute (missing handler for input SB_dns)
> > > > > > > took
> > > > > > > 8764ms
> > > > > > > 2024-05-08T15:14:53.989Z|30732|inc_proc_eng|INFO|node:
> > > > > > > logical_flow_output, recompute (missing handler for input SB_dns)
> > > > > > > took
> > > > > > > 8802ms
> > > > > > > 2024-05-08T15:16:30.911Z|30757|inc_proc_eng|INFO|node:
> > > > > > > logical_flow_output, recompute (missing handler for input SB_dns)
> > > > > > > took
> > > > > > > 8776ms
> > > > > > > 2024-05-08T15:17:09.371Z|30784|inc_proc_eng|INFO|node:
> > > > > > > logical_flow_output, recompute (missing handler for input SB_dns)
> > > > > > > took
> > > > > > > 9062ms
> > > > > > >
> > > > > > > Why would this happen and is there something I can do about it ?
> > > > > > > Are
> > > > > > > there more logs needed ?
> > > > > >
> > > > > > This indicates that your deployment is creating, updating or
> > > > > > deleting a DNS row
> > > > > > in the Northbound database and in turn ovn-northd is updating the
> > > > > > SB DNS rows.
> > > > > > When ovn-controller receives the Southbound DNS updates, it falls
> > > > > > back
> > > > > > to a full recompute
> > > > > > because we are not handling these changes incrementally. Since OVN
> > > > > > native DNS is configured
> > > > > > in your deployment, each DNS packet is sent to ovn-controller for
> > > > > > lookup.
> > > > > > Even though a separate pinctrl thread handles packet-ins, dns
> > > > > > handling is blocked until
> > > > > > the main ovn-controller thread releases a mutex [1].
> > > > > >
> > > > > > There are a few ways to resolve this
> > > > > >
> > > > > > 1. Disable native OVN DNS if you're not using this feature. To
> > > > > > disable, don't create any DNS records in the OVN Northbound db.
> > > > > > 2. Investigate why your deployment is updating the NB DBS table and
> > > > > > avoid it if its not required.
> > > > > > 3. Implement a handler for SB DNS so that ovn-controller do not
> > > > > > fall
> > > > > > back to a full recompute
> > > > > > 4. Avoid locking on the mutex for DNS handling in pinctrl thread
> > > > > > [1].
> > > > > >
> > > > > > (3) or (4) requires code changes.
> > > > > >
> > > > > > Thanks
> > > > > > Numan
> > > > > >
> > > > > > [1] -
> > > > > > https://github.com/ovn-org/ovn/blob/main/controller/pinctrl.c#L3807
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > Gav
> > > > > > > _______________________________________________
> > > > > > > discuss mailing list
> > > > > > > disc...@openvswitch.org
> > > > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> > > > > > >
> > > > _______________________________________________
> > > > discuss mailing list
> > > > disc...@openvswitch.org
> > > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> _______________________________________________
> discuss mailing list
> disc...@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
_______________________________________________
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
