Batch Inference System

{ "type": "excalidraw", "version": 2, "source": "https://marketplace.visualstudio.com/items?itemName=pomdtr.excalidraw-editor", "elements": [ { "id": "NUk1186xEA0GZUSLLrzql", "type": "text", "x": 60.85546875, "y": -231.375, "width": 819.3993530273438, "height": 150, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a2", "roundness": null, "seed": 966015091, "version": 247, "versionNonce": 940726141, "isDeleted": false, "boundElements": null, "updated": 1774382894593, "link": null, "locked": false, "text": "FR\n\n- Synchronous request: client sends POST, waits, gets response on same connection\n- Internal batching: request grouped invisibly client never sees this\n- Tier-based priority: Enterprise first, Paid second, free users last\n", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "FR\n\n- Synchronous request: client sends POST, waits, gets response on same connection\n- Internal batching: request grouped invisibly client never sees this\n- Tier-based priority: Enterprise first, Paid second, free users last\n", "autoResize": true, "lineHeight": 1.25 }, { "id": "pKtrCIPYRkOlY75YSS9rS", "type": "text", "x": 1020, "y": -266.36328125, "width": 447.0396728515625, "height": 175, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a3", "roundness": null, "seed": 239900595, "version": 280, "versionNonce": 466699283, "isDeleted": false, "boundElements": null, "updated": 1774383164482, "link": null, "locked": false, "text": "NFR\n\n- Latency: P95 under 500ms end to end\n- Throughput: 1k RPS now / 10k RPS target\n- Availability 99.9% uptime / no silent failures\n- GPU until 70%, never 100%\n- Backpressure: return 429 early", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "NFR\n\n- Latency: P95 under 500ms end to end\n- Throughput: 1k RPS now / 10k RPS target\n- Availability 99.9% uptime / no silent failures\n- GPU until 70%, never 100%\n- Backpressure: return 429 early", "autoResize": true, "lineHeight": 1.25 }, { "id": "Kbsh0mTv_i3RTZhT0QDl2", "type": "text", "x": 80.92578125, "y": -49.62890625, "width": 588.2395629882812, "height": 475, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a4", "roundness": null, "seed": 60716573, "version": 451, "versionNonce": 1184156147, "isDeleted": false, "boundElements": null, "updated": 1774383166964, "link": null, "locked": false, "text": "API\n\nclient API\n\nPOST /v1/inference\nInput: mode;, input string\nOutput: request_id, output, latency_ms\n\nclient holds connection open, same call returns the response.\nNo polling needed\n\nInternal API\n\nPOST /internal/batch\ninput: batch_id, list of requests\noutput: results per request, status\n\nInternal only. clients never see this.\nGPU maps outputs by index position not by request ID.", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "API\n\nclient API\n\nPOST /v1/inference\nInput: mode;, input string\nOutput: request_id, output, latency_ms\n\nclient holds connection open, same call returns the response.\nNo polling needed\n\nInternal API\n\nPOST /internal/batch\ninput: batch_id, list of requests\noutput: results per request, status\n\nInternal only. clients never see this.\nGPU maps outputs by index position not by request ID.", "autoResize": true, "lineHeight": 1.25 }, { "id": "E84TP3WCY_ucqUHo-EGsY", "type": "text", "x": 1052.62109375, "y": 11.36328125, "width": 766.8394775390625, "height": 700, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a5", "roundness": null, "seed": 498956157, "version": 634, "versionNonce": 2031734867, "isDeleted": false, "boundElements": null, "updated": 1774383377350, "link": null, "locked": false, "text": "GPU Architecture\n\nConstraint 1:\n- one batch at a time per GPU\n\nC2:\n- Inside a batch everthing runs in parallel\n\nC3:\nNVIDIA/CUDA\n- inputs pinned to memory slot, outputs always=result for input\n\n\nCUDA threads execute in warps of 32 - sequential memory is the key\n\nthread i reads: base + i x stride\ni writes: out_base+ixstride\nno parsing. no lookups\n\nID based routing= random\nmemory access= 10-20x slower\ndestroys GPU parallelism\n\nclean division\nGPU: arrays+indices only extremely fast\nBatcher(CPU): holds index-to-user map resolves by psotion after GPU returns\n\n", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "GPU Architecture\n\nConstraint 1:\n- one batch at a time per GPU\n\nC2:\n- Inside a batch everthing runs in parallel\n\nC3:\nNVIDIA/CUDA\n- inputs pinned to memory slot, outputs always=result for input\n\n\nCUDA threads execute in warps of 32 - sequential memory is the key\n\nthread i reads: base + i x stride\ni writes: out_base+ixstride\nno parsing. no lookups\n\nID based routing= random\nmemory access= 10-20x slower\ndestroys GPU parallelism\n\nclean division\nGPU: arrays+indices only extremely fast\nBatcher(CPU): holds index-to-user map resolves by psotion after GPU returns\n\n", "autoResize": true, "lineHeight": 1.25 }, { "id": "TDxWzpZ6ROlNRc4eWGSub", "type": "rectangle", "x": 1053.4375, "y": 790.671875, "width": 233.671875, "height": 141.796875, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a7", "roundness": { "type": 3 }, "seed": 53361779, "version": 43, "versionNonce": 1246574835, "isDeleted": false, "boundElements": [ { "type": "text", "id": "9Yx5M6FiJrbWv_aCUl6kP" }, { "id": "3ZhCkks3Ad2_NMcmA5Aad", "type": "arrow" } ], "updated": 1774383458295, "link": null, "locked": false }, { "id": "9Yx5M6FiJrbWv_aCUl6kP", "type": "text", "x": 1083.4735107421875, "y": 836.5703125, "width": 173.599853515625, "height": 50, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a7V", "roundness": null, "seed": 701057373, "version": 29, "versionNonce": 724789309, "isDeleted": false, "boundElements": null, "updated": 1774383412418, "link": null, "locked": false, "text": "CPU RAM\nbtachstring called", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "TDxWzpZ6ROlNRc4eWGSub", "originalText": "CPU RAM\nbtachstring called", "autoResize": true, "lineHeight": 1.25 }, { "id": "eFzzPiIrrBpOdEPA7s3nF", "type": "rectangle", "x": 1543.03125, "y": 791.671875, "width": 178.99609375, "height": 137.77734375, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a8", "roundness": { "type": 3 }, "seed": 1447434269, "version": 46, "versionNonce": 1475666291, "isDeleted": false, "boundElements": [ { "type": "text", "id": "MImj9RG5zSnDtrstxa3f4" }, { "id": "3ZhCkks3Ad2_NMcmA5Aad", "type": "arrow" }, { "id": "2lSERQCbjTKTiRYgjzhp9", "type": "arrow" } ], "updated": 1774383461083, "link": null, "locked": false }, { "id": "MImj9RG5zSnDtrstxa3f4", "type": "text", "x": 1559.2693481445312, "y": 835.560546875, "width": 146.5198974609375, "height": 50, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a8V", "roundness": null, "seed": 45210963, "version": 27, "versionNonce": 717025587, "isDeleted": false, "boundElements": null, "updated": 1774383420310, "link": null, "locked": false, "text": "GPU RAM\nloads all inputs", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "eFzzPiIrrBpOdEPA7s3nF", "originalText": "GPU RAM\nloads all inputs", "autoResize": true, "lineHeight": 1.25 }, { "id": "jqh908ScWYCTx2IwzjTJ8", "type": "rectangle", "x": 1567.25, "y": 1028.3125, "width": 203.203125, "height": 149.609375, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a9", "roundness": { "type": 3 }, "seed": 1730870259, "version": 41, "versionNonce": 67145875, "isDeleted": false, "boundElements": [ { "type": "text", "id": "pZtGdh3KYiRy8nNU-ezFP" }, { "id": "2lSERQCbjTKTiRYgjzhp9", "type": "arrow" }, { "id": "S-0YckhPunLvDjyKhvIbE", "type": "arrow" } ], "updated": 1774383465161, "link": null, "locked": false }, { "id": "pZtGdh3KYiRy8nNU-ezFP", "type": "text", "x": 1609.1116104125977, "y": 1078.1171875, "width": 119.47990417480469, "height": 50, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a9V", "roundness": null, "seed": 1847410611, "version": 28, "versionNonce": 1068232211, "isDeleted": false, "boundElements": null, "updated": 1774383436509, "link": null, "locked": false, "text": "CUDA kernal\n100ms fixed", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "jqh908ScWYCTx2IwzjTJ8", "originalText": "CUDA kernal\n100ms fixed", "autoResize": true, "lineHeight": 1.25 }, { "id": "3ooIXrhUz_RjsPSro3S_z", "type": "rectangle", "x": 1066.46875, "y": 1035.765625, "width": 247.515625, "height": 147.71875, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aA", "roundness": { "type": 3 }, "seed": 2146641011, "version": 87, "versionNonce": 1236125651, "isDeleted": false, "boundElements": [ { "type": "text", "id": "o3nw85QrNM_pDPm9oixeN" }, { "id": "S-0YckhPunLvDjyKhvIbE", "type": "arrow" } ], "updated": 1774383465161, "link": null, "locked": false }, { "id": "o3nw85QrNM_pDPm9oixeN", "type": "text", "x": 1074.1266403198242, "y": 1072.125, "width": 232.19984436035156, "height": 75, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aB", "roundness": null, "seed": 271874493, "version": 69, "versionNonce": 1731177363, "isDeleted": false, "boundElements": null, "updated": 1774383454162, "link": null, "locked": false, "text": "CPU RAM\nresults returned resolve\nby index", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "3ooIXrhUz_RjsPSro3S_z", "originalText": "CPU RAM\nresults returned resolve by index", "autoResize": true, "lineHeight": 1.25 }, { "id": "3ZhCkks3Ad2_NMcmA5Aad", "type": "arrow", "x": 1285.09375, "y": 859.15625, "width": 253.9375, "height": 4.953125, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aC", "roundness": { "type": 2 }, "seed": 687038333, "version": 42, "versionNonce": 1354736275, "isDeleted": false, "boundElements": null, "updated": 1774383458295, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 253.9375, -4.953125 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "TDxWzpZ6ROlNRc4eWGSub", "focus": -0.0023840546429043956, "gap": 2.015625 }, "endBinding": { "elementId": "eFzzPiIrrBpOdEPA7s3nF", "focus": 0.1158236423722208, "gap": 4 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "2lSERQCbjTKTiRYgjzhp9", "type": "arrow", "x": 1637.828125, "y": 935.296875, "width": 1.078125, "height": 96.015625, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aD", "roundness": { "type": 2 }, "seed": 1675174109, "version": 32, "versionNonce": 207502099, "isDeleted": false, "boundElements": null, "updated": 1774383461083, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 1.078125, 96.015625 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "eFzzPiIrrBpOdEPA7s3nF", "focus": -0.049402492536741, "gap": 5.84765625 }, "endBinding": { "elementId": "jqh908ScWYCTx2IwzjTJ8", "focus": -0.2844456441340154, "gap": 3 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "S-0YckhPunLvDjyKhvIbE", "type": "arrow", "x": 1567.1875, "y": 1101.09375, "width": 248.828125, "height": 0.484375, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aE", "roundness": { "type": 2 }, "seed": 310931549, "version": 50, "versionNonce": 359073331, "isDeleted": false, "boundElements": null, "updated": 1774383465161, "link": null, "locked": false, "points": [ [ 0, 0 ], [ -248.828125, 0.484375 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "jqh908ScWYCTx2IwzjTJ8", "focus": 0.029616882699372033, "gap": 1 }, "endBinding": { "elementId": "3ooIXrhUz_RjsPSro3S_z", "focus": -0.10522831978473754, "gap": 4.375 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "UCfSj4Q4m1FBcOkwH9UHI", "type": "text", "x": 1386, "y": 837, "width": 92.65992736816406, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aF", "roundness": null, "seed": 396771229, "version": 10, "versionNonce": 1929688829, "isDeleted": false, "boundElements": null, "updated": 1774383470739, "link": null, "locked": false, "text": "PCIe 5ms", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "PCIe 5ms", "autoResize": true, "lineHeight": 1.25 }, { "id": "wkve_32nPqAP1X14wi7X1", "type": "text", "x": 1661, "y": 970, "width": 59.219970703125, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aG", "roundness": null, "seed": 16433203, "version": 7, "versionNonce": 1859632627, "isDeleted": false, "boundElements": null, "updated": 1774383474217, "link": null, "locked": false, "text": "100ms", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "100ms", "autoResize": true, "lineHeight": 1.25 }, { "id": "0DSBArHolwuKTmUakNy_c", "type": "text", "x": 1416, "y": 1076, "width": 92.65992736816406, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aH", "roundness": null, "seed": 1982907261, "version": 12, "versionNonce": 521636765, "isDeleted": false, "boundElements": null, "updated": 1774383480444, "link": null, "locked": false, "text": "PCIe 5ms", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "PCIe 5ms", "autoResize": true, "lineHeight": 1.25 }, { "id": "1dXOSku8x7-A91Kr-B0d-", "type": "text", "x": 1300, "y": 1244.875, "width": 280.71978759765625, "height": 50, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aI", "roundness": null, "seed": 1634107795, "version": 62, "versionNonce": 934341917, "isDeleted": false, "boundElements": null, "updated": 1774383511209, "link": null, "locked": false, "text": "Total: ` 110ms every time\n1 string or 100 strings. same", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Total: ` 110ms every time\n1 string or 100 strings. same", "autoResize": true, "lineHeight": 1.25 }, { "id": "PMKInKb64-IEByrEbvOGo", "type": "rectangle", "x": 1895.3124999999998, "y": -3.1015625, "width": 234.734375, "height": 160.015625, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aJ", "roundness": { "type": 3 }, "seed": 495384221, "version": 302, "versionNonce": 1605703603, "isDeleted": false, "boundElements": [ { "id": "07uLIQMAQtYaRoO9xuLZK", "type": "arrow" }, { "id": "klH4e-i7X1h1HZt7LAGJr", "type": "arrow" }, { "type": "text", "id": "4rk3ekdPix18htzOiyn5E" } ], "updated": 1774384669906, "link": null, "locked": false }, { "id": "4rk3ekdPix18htzOiyn5E", "type": "text", "x": 1925.2897415161133, "y": 1.90625, "width": 174.77989196777344, "height": 150, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aJV", "roundness": null, "seed": 398999261, "version": 353, "versionNonce": 322808147, "isDeleted": false, "boundElements": null, "updated": 1774384669906, "link": null, "locked": false, "text": "One batch cycle\nassembly 40 ms\nPCIe in 5ms\nGPU kernel 100ms\nPCIe out 5ms\ntotal 150ms", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "PMKInKb64-IEByrEbvOGo", "originalText": "One batch cycle\nassembly 40 ms\nPCIe in 5ms\nGPU kernel 100ms\nPCIe out 5ms\ntotal 150ms", "autoResize": true, "lineHeight": 1.25 }, { "id": "5KGH82gwagU9858THIkIO", "type": "rectangle", "x": 2330.640625, "y": -4.9765625, "width": 214.65625, "height": 152.7421875, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aK", "roundness": { "type": 3 }, "seed": 1567200851, "version": 221, "versionNonce": 1418602035, "isDeleted": false, "boundElements": [ { "id": "07uLIQMAQtYaRoO9xuLZK", "type": "arrow" }, { "type": "text", "id": "xkXlcTyVubpDz8SPA7wV6" } ], "updated": 1774384669906, "link": null, "locked": false }, { "id": "xkXlcTyVubpDz8SPA7wV6", "type": "text", "x": 2339.658805847168, "y": 8.89453125, "width": 196.61988830566406, "height": 125, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aKV", "roundness": null, "seed": 903881139, "version": 254, "versionNonce": 1458628563, "isDeleted": false, "boundElements": null, "updated": 1774384669906, "link": null, "locked": false, "text": "GPUs needed\n1k RPS raw: 1.5\nGPUs\nat 70% util 3 GPUs\n10k RPS: `28 GPUs", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "5KGH82gwagU9858THIkIO", "originalText": "GPUs needed\n1k RPS raw: 1.5 GPUs\nat 70% util 3 GPUs\n10k RPS: `28 GPUs", "autoResize": true, "lineHeight": 1.25 }, { "id": "UBt7rQqQatSa7Fy-6S2Ia", "type": "rectangle", "x": 1894.3281249999998, "y": 228.1171875, "width": 259.8125, "height": 152.03125, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aL", "roundness": { "type": 3 }, "seed": 777354355, "version": 152, "versionNonce": 459362067, "isDeleted": false, "boundElements": [ { "id": "klH4e-i7X1h1HZt7LAGJr", "type": "arrow" }, { "id": "8jxerlQcfd7krCd0IVkT7", "type": "arrow" }, { "type": "text", "id": "pWca-D2ShsW5Qk2dtAKtj" } ], "updated": 1774384669906, "link": null, "locked": false }, { "id": "pWca-D2ShsW5Qk2dtAKtj", "type": "text", "x": 1906.774444580078, "y": 254.1328125, "width": 234.91986083984375, "height": 100, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aLV", "roundness": null, "seed": 515251549, "version": 180, "versionNonce": 791409843, "isDeleted": false, "boundElements": null, "updated": 1774384669906, "link": null, "locked": false, "text": "Per GPU per second\n1000/150 = 6.7 batches\nx 100 requests\n= 670 req/sec/GPU", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "UBt7rQqQatSa7Fy-6S2Ia", "originalText": "Per GPU per second\n1000/150 = 6.7 batches\nx 100 requests\n= 670 req/sec/GPU", "autoResize": true, "lineHeight": 1.25 }, { "id": "BoBd8Ms1tFpP87XyCeI6v", "type": "rectangle", "x": 2307.04296875, "y": 225.5625, "width": 251.41796875, "height": 160, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aM", "roundness": { "type": 3 }, "seed": 418120787, "version": 206, "versionNonce": 1345564051, "isDeleted": false, "boundElements": [ { "id": "8jxerlQcfd7krCd0IVkT7", "type": "arrow" }, { "type": "text", "id": "vYBUVHFXzMh8MfTFE2El-" } ], "updated": 1774384669906, "link": null, "locked": false }, { "id": "vYBUVHFXzMh8MfTFE2El-", "type": "text", "x": 2313.222023010254, "y": 230.5625, "width": 239.0598602294922, "height": 150, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aMV", "roundness": null, "seed": 435841021, "version": 249, "versionNonce": 120488755, "isDeleted": false, "boundElements": null, "updated": 1774384669906, "link": null, "locked": false, "text": "Why 70% not 100%\nGPU cold start = 5 mins\nneed headroom for\nspikes\n30% buffer= stable\nlatency", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "BoBd8Ms1tFpP87XyCeI6v", "originalText": "Why 70% not 100%\nGPU cold start = 5 mins\nneed headroom for spikes\n30% buffer= stable latency", "autoResize": true, "lineHeight": 1.25 }, { "id": "07uLIQMAQtYaRoO9xuLZK", "type": "arrow", "x": 2133.147190335924, "y": 67.76258396223875, "width": 193.67704384404033, "height": 0.8723674563711938, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aN", "roundness": { "type": 2 }, "seed": 1131502909, "version": 551, "versionNonce": 34192317, "isDeleted": false, "boundElements": null, "updated": 1774384669914, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 193.67704384404033, -0.8723674563711938 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "PMKInKb64-IEByrEbvOGo", "focus": -0.10679699013648968, "gap": 3.100315335923824 }, "endBinding": { "elementId": "5KGH82gwagU9858THIkIO", "focus": 0.06512220407293186, "gap": 3.816390820035849 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "klH4e-i7X1h1HZt7LAGJr", "type": "arrow", "x": 2014.201447618454, "y": 161.44657911403374, "width": 1.1225137607784745, "height": 62.885416031507475, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aO", "roundness": { "type": 2 }, "seed": 1158237427, "version": 469, "versionNonce": 835825789, "isDeleted": false, "boundElements": null, "updated": 1774384669914, "link": null, "locked": false, "points": [ [ 0, 0 ], [ -1.1225137607784745, 62.885416031507475 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "PMKInKb64-IEByrEbvOGo", "focus": -0.025512930423624583, "gap": 4.532516614033739 }, "endBinding": { "elementId": "UBt7rQqQatSa7Fy-6S2Ia", "focus": -0.09583724167447144, "gap": 3.785192354458786 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "8jxerlQcfd7krCd0IVkT7", "type": "arrow", "x": 2157.494242604651, "y": 300.5105000109604, "width": 145.2529469515, "height": 1.4584909585905734, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aP", "roundness": { "type": 2 }, "seed": 270370269, "version": 355, "versionNonce": 1915974973, "isDeleted": false, "boundElements": null, "updated": 1774384669914, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 145.2529469515, 1.4584909585905734 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "UBt7rQqQatSa7Fy-6S2Ia", "focus": -0.06415389557169054, "gap": 3.3536176046509354 }, "endBinding": { "elementId": "BoBd8Ms1tFpP87XyCeI6v", "focus": 0.028157302385824496, "gap": 4.29577919384883 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "1Wc62qXC1WZMiiJ5TTNAg", "type": "rectangle", "x": 429.1041666666663, "y": 1608.4375000000005, "width": 166.671875, "height": 60.2734375, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aS", "roundness": { "type": 3 }, "seed": 1038861725, "version": 106, "versionNonce": 1059181533, "isDeleted": false, "boundElements": [ { "type": "text", "id": "VsPQO_JTF34e63e6LFHYc" }, { "id": "N1F_22O4jruaZjukq0GOC", "type": "arrow" }, { "id": "wGAHu9wKl_noIJY01qU9_", "type": "arrow" } ], "updated": 1774384870045, "link": null, "locked": false }, { "id": "VsPQO_JTF34e63e6LFHYc", "type": "text", "x": 474.5401331583655, "y": 1626.0742187500005, "width": 75.79994201660156, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aT", "roundness": null, "seed": 907795027, "version": 86, "versionNonce": 526074813, "isDeleted": false, "boundElements": null, "updated": 1774384763444, "link": null, "locked": false, "text": "Client A", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "1Wc62qXC1WZMiiJ5TTNAg", "originalText": "Client A", "autoResize": true, "lineHeight": 1.25 }, { "id": "53DQ7_DKVNW-Bd_z91s1B", "type": "rectangle", "x": 701.3854166666663, "y": 1587.2500000000005, "width": 178.578125, "height": 94.94531249999999, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aU", "roundness": { "type": 3 }, "seed": 1405801981, "version": 105, "versionNonce": 964964381, "isDeleted": false, "boundElements": [ { "type": "text", "id": "uLRrX1954n-8GOvyNCOlY" }, { "id": "wGAHu9wKl_noIJY01qU9_", "type": "arrow" }, { "id": "oE75WqUKZhSeslEZpA9-d", "type": "arrow" } ], "updated": 1774384872358, "link": null, "locked": false }, { "id": "uLRrX1954n-8GOvyNCOlY", "type": "text", "x": 722.2645365397132, "y": 1622.2226562500005, "width": 136.81988525390625, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aUV", "roundness": null, "seed": 695846525, "version": 85, "versionNonce": 1526187677, "isDeleted": false, "boundElements": null, "updated": 1774384764363, "link": null, "locked": false, "text": "Load Balancer", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "53DQ7_DKVNW-Bd_z91s1B", "originalText": "Load Balancer", "autoResize": true, "lineHeight": 1.25 }, { "id": "7x_GQ8E_N8rUkN0ETTc4V", "type": "rectangle", "x": 998.7291666666663, "y": 1573.9062500000005, "width": 181.6953125, "height": 141.2578125, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aV", "roundness": { "type": 3 }, "seed": 1858221149, "version": 135, "versionNonce": 2145422355, "isDeleted": false, "boundElements": [ { "type": "text", "id": "-PiU32-PtngZmMLcsh_B4" }, { "id": "oE75WqUKZhSeslEZpA9-d", "type": "arrow" }, { "id": "K6Qj5_DINlHZmegzl-jVt", "type": "arrow" } ], "updated": 1774384877211, "link": null, "locked": false }, { "id": "-PiU32-PtngZmMLcsh_B4", "type": "text", "x": 1024.2368723551429, "y": 1607.0351562500005, "width": 130.67990112304688, "height": 75, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aVV", "roundness": null, "seed": 1963979635, "version": 116, "versionNonce": 1228563005, "isDeleted": false, "boundElements": null, "updated": 1774384765577, "link": null, "locked": false, "text": "API Gateway\nauth+tier\n429 if full", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "7x_GQ8E_N8rUkN0ETTc4V", "originalText": "API Gateway\nauth+tier\n429 if full", "autoResize": true, "lineHeight": 1.25 }, { "id": "QDErWrhRoXqgI8pP0ITYQ", "type": "rectangle", "x": 982.3072916666663, "y": 1877.4062500000005, "width": 279.6328125000001, "height": 68.2265625, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aW", "roundness": { "type": 3 }, "seed": 1773126611, "version": 183, "versionNonce": 1773853917, "isDeleted": false, "boundElements": [ { "type": "text", "id": "sNeu6ysSmUj6lQrkuH25t" }, { "id": "K6Qj5_DINlHZmegzl-jVt", "type": "arrow" }, { "id": "C5ByaQT4yKnYT544JRMc6", "type": "arrow" } ], "updated": 1774384982555, "link": null, "locked": false }, { "id": "sNeu6ysSmUj6lQrkuH25t", "type": "text", "x": 1040.753763834635, "y": 1899.0195312500005, "width": 162.7398681640625, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aWV", "roundness": null, "seed": 1154872285, "version": 162, "versionNonce": 191183165, "isDeleted": false, "boundElements": null, "updated": 1774384982555, "link": null, "locked": false, "text": "Enterprise queue", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "QDErWrhRoXqgI8pP0ITYQ", "originalText": "Enterprise queue", "autoResize": true, "lineHeight": 1.25 }, { "id": "dJwRimV9z1kG82SFWZqe0", "type": "rectangle", "x": 1435.7838541666663, "y": 1934.3515625000005, "width": 246.76562499999977, "height": 164.640625, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aX", "roundness": { "type": 3 }, "seed": 469174963, "version": 150, "versionNonce": 1311010323, "isDeleted": false, "boundElements": [ { "type": "text", "id": "g85baapPHOuXsG_V3c8jp" }, { "id": "C5ByaQT4yKnYT544JRMc6", "type": "arrow" }, { "id": "YuI05eBVl3wPWbU4KuAHG", "type": "arrow" }, { "id": "ujc9fSry-S6s7lkxHsEm5", "type": "arrow" }, { "id": "IpEkf8tvboZ5INAPG_QVe", "type": "arrow" } ], "updated": 1774384889380, "link": null, "locked": false }, { "id": "g85baapPHOuXsG_V3c8jp", "type": "text", "x": 1471.5167032877598, "y": 1979.1718750000005, "width": 175.2999267578125, "height": 75, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aXV", "roundness": null, "seed": 1754744883, "version": 76, "versionNonce": 992864957, "isDeleted": false, "boundElements": null, "updated": 1774384817058, "link": null, "locked": false, "text": "Request Batcher\nsize = 100\n or 40 ms timeout", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "dJwRimV9z1kG82SFWZqe0", "originalText": "Request Batcher\nsize = 100\n or 40 ms timeout", "autoResize": true, "lineHeight": 1.25 }, { "id": "r7Bn73_lGMyoY4vyWgU5V", "type": "rectangle", "x": 2705.104166666666, "y": 2152.5312500000005, "width": 667.484375, "height": 465.9765625, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aY", "roundness": { "type": 3 }, "seed": 660698323, "version": 41, "versionNonce": 2070807901, "isDeleted": false, "boundElements": null, "updated": 1774384708814, "link": null, "locked": false }, { "id": "mAi3F7-CUhziNgGodsLCx", "type": "rectangle", "x": 980.9596354166663, "y": 1977.1054687500005, "width": 279.6328125000001, "height": 68.2265625, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aZ", "roundness": { "type": 3 }, "seed": 1271107059, "version": 217, "versionNonce": 1061177469, "isDeleted": false, "boundElements": [ { "type": "text", "id": "JxojtlPi4cDtUxCRaa94R" }, { "id": "YuI05eBVl3wPWbU4KuAHG", "type": "arrow" } ], "updated": 1774384968567, "link": null, "locked": false }, { "id": "JxojtlPi4cDtUxCRaa94R", "type": "text", "x": 1068.8560816446936, "y": 1998.7187500000005, "width": 103.83992004394531, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aa", "roundness": null, "seed": 1520654227, "version": 211, "versionNonce": 1350316253, "isDeleted": false, "boundElements": [], "updated": 1774384968567, "link": null, "locked": false, "text": "Paid queue", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "mAi3F7-CUhziNgGodsLCx", "originalText": "Paid queue", "autoResize": true, "lineHeight": 1.25 }, { "id": "XOwoNh5vm5Fy3lvs0u5lT", "type": "rectangle", "x": 969.7955729166663, "y": 2070.9804687500005, "width": 279.6328125000001, "height": 68.2265625, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ab", "roundness": { "type": 3 }, "seed": 1485502525, "version": 162, "versionNonce": 950109629, "isDeleted": false, "boundElements": [ { "type": "text", "id": "d5ZOIgifZmbRTilT7lqQP" }, { "id": "ujc9fSry-S6s7lkxHsEm5", "type": "arrow" } ], "updated": 1774384969684, "link": null, "locked": false }, { "id": "d5ZOIgifZmbRTilT7lqQP", "type": "text", "x": 1058.8920160929358, "y": 2092.5937500000005, "width": 101.43992614746094, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ac", "roundness": null, "seed": 341502109, "version": 156, "versionNonce": 490980893, "isDeleted": false, "boundElements": [], "updated": 1774384969684, "link": null, "locked": false, "text": "free queue", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "XOwoNh5vm5Fy3lvs0u5lT", "originalText": "free queue", "autoResize": true, "lineHeight": 1.25 }, { "id": "DhHvPverX9pT3WfTHpnP_", "type": "rectangle", "x": 1821.1979166666663, "y": 2199.5781250000005, "width": 320.15624999999966, "height": 98.171875, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ad", "roundness": { "type": 3 }, "seed": 1156980605, "version": 90, "versionNonce": 427749149, "isDeleted": false, "boundElements": [ { "type": "text", "id": "TIEOXKLZN3B0tIPvDtUQ1" }, { "id": "IpEkf8tvboZ5INAPG_QVe", "type": "arrow" }, { "id": "9rxkk3IL_Q-UZyjqk3XVI", "type": "arrow" } ], "updated": 1774384975406, "link": null, "locked": false }, { "id": "TIEOXKLZN3B0tIPvDtUQ1", "type": "text", "x": 1898.3960901896153, "y": 2236.1640625000005, "width": 165.75990295410156, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ae", "roundness": null, "seed": 101750867, "version": 57, "versionNonce": 2136344637, "isDeleted": false, "boundElements": null, "updated": 1774384975408, "link": null, "locked": false, "text": "GPU worker Pool", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "DhHvPverX9pT3WfTHpnP_", "originalText": "GPU worker Pool", "autoResize": true, "lineHeight": 1.25 }, { "id": "vPkfVn9fPCZJlypMKIYJf", "type": "rectangle", "x": 1283.3229166666663, "y": 2453.8437500000005, "width": 420.3125, "height": 107.8125, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "af", "roundness": { "type": 3 }, "seed": 890083805, "version": 71, "versionNonce": 1054139667, "isDeleted": false, "boundElements": [ { "type": "text", "id": "Mqdzj6Z3oYzZEe-a9ze-w" }, { "id": "N1F_22O4jruaZjukq0GOC", "type": "arrow" }, { "id": "9rxkk3IL_Q-UZyjqk3XVI", "type": "arrow" } ], "updated": 1774384893416, "link": null, "locked": false }, { "id": "Mqdzj6Z3oYzZEe-a9ze-w", "type": "text", "x": 1350.5792643229163, "y": 2482.7500000000005, "width": 285.7998046875, "height": 50, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ag", "roundness": null, "seed": 494505821, "version": 48, "versionNonce": 480969053, "isDeleted": false, "boundElements": null, "updated": 1774384854059, "link": null, "locked": false, "text": "Response Handler:\noutputs[i]->future[i]->HTTP", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "vPkfVn9fPCZJlypMKIYJf", "originalText": "Response Handler:\noutputs[i]->future[i]->HTTP", "autoResize": true, "lineHeight": 1.25 }, { "id": "N1F_22O4jruaZjukq0GOC", "type": "arrow", "x": 1280.3307291666663, "y": 2503.4296875000005, "width": 774.625, "height": 831, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ah", "roundness": { "type": 2 }, "seed": 1406537245, "version": 191, "versionNonce": 663886067, "isDeleted": false, "boundElements": null, "updated": 1774384866790, "link": null, "locked": false, "points": [ [ 0, 0 ], [ -661.9921875, -186.703125 ], [ -774.625, -831 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "vPkfVn9fPCZJlypMKIYJf", "focus": -0.49298331842124893, "gap": 2.9921875 }, "endBinding": { "elementId": "1Wc62qXC1WZMiiJ5TTNAg", "focus": 0.14280145518052292, "gap": 3.71875 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "wGAHu9wKl_noIJY01qU9_", "type": "arrow", "x": 603.5416666666663, "y": 1638.4375000000005, "width": 89.703125, "height": 0.3203125, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ai", "roundness": { "type": 2 }, "seed": 1919451795, "version": 41, "versionNonce": 1817752659, "isDeleted": false, "boundElements": null, "updated": 1774384965583, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 89.703125, 0.3203125 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "1Wc62qXC1WZMiiJ5TTNAg", "focus": -0.015181057941584914, "gap": 7.765625 }, "endBinding": { "elementId": "53DQ7_DKVNW-Bd_z91s1B", "focus": -0.09171211841086575, "gap": 8.140625 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "oE75WqUKZhSeslEZpA9-d", "type": "arrow", "x": 886.6041666666663, "y": 1640.6875000000005, "width": 107.46875, "height": 4.1328125, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aj", "roundness": { "type": 2 }, "seed": 2134668627, "version": 36, "versionNonce": 354559101, "isDeleted": false, "boundElements": null, "updated": 1774384872359, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 107.46875, -4.1328125 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "53DQ7_DKVNW-Bd_z91s1B", "focus": 0.1896405922845937, "gap": 6.640625 }, "endBinding": { "elementId": "7x_GQ8E_N8rUkN0ETTc4V", "focus": 0.1572148234322314, "gap": 4.65625 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "K6Qj5_DINlHZmegzl-jVt", "type": "arrow", "x": 1099.4579487926273, "y": 1723.2130297084411, "width": 6.242758733995515, "height": 152.68355743141228, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ak", "roundness": { "type": 2 }, "seed": 322245907, "version": 88, "versionNonce": 1413810589, "isDeleted": false, "boundElements": null, "updated": 1774384982555, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 6.242758733995515, 152.68355743141228 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "7x_GQ8E_N8rUkN0ETTc4V", "focus": -0.06599381726684228, "gap": 12.8359375 }, "endBinding": { "elementId": "QDErWrhRoXqgI8pP0ITYQ", "focus": -0.10703972546560162, "gap": 5.71875 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "C5ByaQT4yKnYT544JRMc6", "type": "arrow", "x": 1276.4642729294028, "y": 1920.9020590157318, "width": 156.9944792443514, "height": 87.16542002333017, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "al", "roundness": { "type": 2 }, "seed": 657456061, "version": 61, "versionNonce": 629919229, "isDeleted": false, "boundElements": null, "updated": 1774384982555, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 156.9944792443514, 87.16542002333017 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "QDErWrhRoXqgI8pP0ITYQ", "focus": -0.6830785609534635, "gap": 15.242187499999773 }, "endBinding": { "elementId": "dJwRimV9z1kG82SFWZqe0", "focus": -0.4054285503457373, "gap": 2.8046875 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "YuI05eBVl3wPWbU4KuAHG", "type": "arrow", "x": 1270.2259330202362, "y": 2013.2627263330783, "width": 159.98133926157607, "height": 0.08476444793518567, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "am", "roundness": { "type": 2 }, "seed": 909753597, "version": 84, "versionNonce": 920390973, "isDeleted": false, "boundElements": null, "updated": 1774384968567, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 159.98133926157607, 0.08476444793518567 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "mAi3F7-CUhziNgGodsLCx", "focus": 0.057427451410874936, "gap": 10.042968749999773 }, "endBinding": { "elementId": "dJwRimV9z1kG82SFWZqe0", "focus": 0.03966420859912659, "gap": 6.7578125 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "ujc9fSry-S6s7lkxHsEm5", "type": "arrow", "x": 1257.1779968691271, "y": 2106.277867753711, "width": 172.60726265833068, "height": 82.23907335193553, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "an", "roundness": { "type": 2 }, "seed": 525107517, "version": 58, "versionNonce": 1216281213, "isDeleted": false, "boundElements": null, "updated": 1774384969684, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 172.60726265833068, -82.23907335193553 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "XOwoNh5vm5Fy3lvs0u5lT", "focus": 0.7097088232042598, "gap": 8.058593749999773 }, "endBinding": { "elementId": "dJwRimV9z1kG82SFWZqe0", "focus": 0.38495974912483977, "gap": 7.2734375 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "IpEkf8tvboZ5INAPG_QVe", "type": "arrow", "x": 1681.0214115138604, "y": 2092.7164876764755, "width": 163.41340764120855, "height": 105.30244400553374, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ao", "roundness": { "type": 2 }, "seed": 364280243, "version": 67, "versionNonce": 1164772221, "isDeleted": false, "boundElements": null, "updated": 1774384975407, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 163.41340764120855, 105.30244400553374 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "dJwRimV9z1kG82SFWZqe0", "focus": -0.015301768731228625, "gap": 6.01092251748026 }, "endBinding": { "elementId": "DhHvPverX9pT3WfTHpnP_", "focus": -0.24656969439796222, "gap": 5.007384992167532 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "9rxkk3IL_Q-UZyjqk3XVI", "type": "arrow", "x": 1849.143898773554, "y": 2299.960696186063, "width": 241.57091566368263, "height": 152.0040477686348, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ap", "roundness": { "type": 2 }, "seed": 1538521533, "version": 74, "versionNonce": 123309021, "isDeleted": false, "boundElements": null, "updated": 1774384975408, "link": null, "locked": false, "points": [ [ 0, 0 ], [ -241.57091566368263, 152.0040477686348 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "DhHvPverX9pT3WfTHpnP_", "focus": 0.21259649234298753, "gap": 6.950408604465784 }, "endBinding": { "elementId": "vPkfVn9fPCZJlypMKIYJf", "focus": 0.08598164026316572, "gap": 6.9609375 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "IGgzo2iQ3EbIhC6mvkxxr", "type": "text", "x": 1461.0729166666663, "y": 1826.9687500000005, "width": 180.59986877441406, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aq", "roundness": null, "seed": 811184925, "version": 64, "versionNonce": 1169068189, "isDeleted": false, "boundElements": null, "updated": 1774384981768, "link": null, "locked": false, "text": "Batch aggregation", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Batch aggregation", "autoResize": true, "lineHeight": 1.25 }, { "id": "-qokyuayQiv_JEVHosqQb", "type": "text", "x": 747.6666666666663, "y": 1702.0000000000005, "width": 89.99993896484375, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ar", "roundness": null, "seed": 896623027, "version": 11, "versionNonce": 1659792883, "isDeleted": false, "boundElements": null, "updated": 1774384913850, "link": null, "locked": false, "text": "sync hold", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "sync hold", "autoResize": true, "lineHeight": 1.25 }, { "id": "0adaH2bhvzCjsmQsmw8iy", "type": "text", "x": 1301.1197916666663, "y": 1909.5234375000005, "width": 140.87989807128906, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "as", "roundness": null, "seed": 1668159549, "version": 34, "versionNonce": 894771539, "isDeleted": false, "boundElements": null, "updated": 1774384925001, "link": null, "locked": false, "text": "poll by priority", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "poll by priority", "autoResize": true, "lineHeight": 1.25 }, { "id": "zcoUZ1d4MERIifYD-_CGP", "type": "text", "x": 1743.2291666666663, "y": 2097.5312500000005, "width": 82.57992553710938, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "at", "roundness": null, "seed": 814587421, "version": 28, "versionNonce": 1615158109, "isDeleted": false, "boundElements": null, "updated": 1774384979021, "link": null, "locked": false, "text": "dispatch", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "dispatch", "autoResize": true, "lineHeight": 1.25 }, { "id": "04YvWW0eFRlsGlYx-tGvg", "type": "text", "x": 1829.6666666666663, "y": 2404.0000000000005, "width": 101.89991760253906, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "au", "roundness": null, "seed": 1690846227, "version": 12, "versionNonce": 107435507, "isDeleted": false, "boundElements": null, "updated": 1774384935406, "link": null, "locked": false, "text": "outputs[i]", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "outputs[i]", "autoResize": true, "lineHeight": 1.25 }, { "id": "12kIfPXtJuKJ7034hvSA6", "type": "text", "x": 1011.4088541666663, "y": 1783.2031250000005, "width": 64.61994934082031, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "av", "roundness": null, "seed": 1957473149, "version": 15, "versionNonce": 1651642579, "isDeleted": false, "boundElements": null, "updated": 1774384942620, "link": null, "locked": false, "text": "by tier", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "by tier", "autoResize": true, "lineHeight": 1.25 }, { "id": "7nquUCxHhnw2cHsy5rV2P", "type": "text", "x": 666.0104166666663, "y": 1552.4062500000005, "width": 229.57984924316406, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "aw", "roundness": null, "seed": 1352746653, "version": 50, "versionNonce": 266132211, "isDeleted": false, "boundElements": null, "updated": 1774384953261, "link": null, "locked": false, "text": "resolve open connection", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "resolve open connection", "autoResize": true, "lineHeight": 1.25 }, { "id": "-l0uMlqcjcn8R0S3z00aM", "type": "rectangle", "x": 929.8229166666663, "y": 2842.8750000000005, "width": 283.40625, "height": 123.53125, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ax", "roundness": { "type": 3 }, "seed": 2062078749, "version": 52, "versionNonce": 1237383869, "isDeleted": false, "boundElements": [ { "type": "text", "id": "6ntmVTobMk2FhiNzBDYtZ" }, { "id": "Da3Q1Z9DLN5ZBRRvigDaG", "type": "arrow" }, { "id": "oRNvZcpqJbmPmch_USP3s", "type": "arrow" } ], "updated": 1774385216905, "link": null, "locked": false }, { "id": "6ntmVTobMk2FhiNzBDYtZ", "type": "text", "x": 1005.156092325846, "y": 2892.1406250000005, "width": 132.73989868164062, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "ay", "roundness": null, "seed": 417472371, "version": 32, "versionNonce": 1835045747, "isDeleted": false, "boundElements": null, "updated": 1774385187502, "link": null, "locked": false, "text": "Shared Queue", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "-l0uMlqcjcn8R0S3z00aM", "originalText": "Shared Queue", "autoResize": true, "lineHeight": 1.25 }, { "id": "XrPldEc-JzLFQ1jwVb2N9", "type": "rectangle", "x": 1476.1666666666663, "y": 2763.6562500000005, "width": 268.3203125, "height": 118.4765625, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "az", "roundness": { "type": 3 }, "seed": 799515037, "version": 47, "versionNonce": 1264908723, "isDeleted": false, "boundElements": [ { "type": "text", "id": "30zMBysWI6whmkDsnMrO3" }, { "id": "Da3Q1Z9DLN5ZBRRvigDaG", "type": "arrow" }, { "id": "lNrUY4Em4xg7GhKPA1NWT", "type": "arrow" } ], "updated": 1774385219568, "link": null, "locked": false }, { "id": "30zMBysWI6whmkDsnMrO3", "type": "text", "x": 1562.956858317057, "y": 2810.3945312500005, "width": 94.73992919921875, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "azV", "roundness": null, "seed": 1395889053, "version": 13, "versionNonce": 298154675, "isDeleted": false, "boundElements": null, "updated": 1774385205115, "link": null, "locked": false, "text": "Batcher 1", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "XrPldEc-JzLFQ1jwVb2N9", "originalText": "Batcher 1", "autoResize": true, "lineHeight": 1.25 }, { "id": "CyQsFhm6jz_-lKxL_iLeV", "type": "rectangle", "x": 1468.6471354166663, "y": 2942.8945312500005, "width": 268.3203125, "height": 118.4765625, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b00", "roundness": { "type": 3 }, "seed": 279374451, "version": 56, "versionNonce": 1209363219, "isDeleted": false, "boundElements": [ { "type": "text", "id": "Z0iTjnZ4pj6f-VupTXFrj" }, { "id": "oRNvZcpqJbmPmch_USP3s", "type": "arrow" }, { "id": "xUHuSXq_wec0CILYpRjnq", "type": "arrow" } ], "updated": 1774385222179, "link": null, "locked": false }, { "id": "Z0iTjnZ4pj6f-VupTXFrj", "type": "text", "x": 1552.7073237101233, "y": 2989.6328125000005, "width": 100.19993591308594, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b01", "roundness": null, "seed": 1116878323, "version": 11, "versionNonce": 1254634451, "isDeleted": false, "boundElements": null, "updated": 1774385198122, "link": null, "locked": false, "text": "Batcher 2", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "CyQsFhm6jz_-lKxL_iLeV", "originalText": "Batcher 2", "autoResize": true, "lineHeight": 1.25 }, { "id": "Xrb-UzhwTn1dzM7CWzaMv", "type": "rectangle", "x": 1952.3190104166658, "y": 2832.6992187500005, "width": 268.3203125, "height": 118.4765625, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b02", "roundness": { "type": 3 }, "seed": 1693934269, "version": 50, "versionNonce": 355336787, "isDeleted": false, "boundElements": [ { "type": "text", "id": "EQxzhN1SUeh8HGaiIEfyg" }, { "id": "lNrUY4Em4xg7GhKPA1NWT", "type": "arrow" }, { "id": "xUHuSXq_wec0CILYpRjnq", "type": "arrow" } ], "updated": 1774385222179, "link": null, "locked": false }, { "id": "EQxzhN1SUeh8HGaiIEfyg", "type": "text", "x": 1995.1692225138338, "y": 2879.4375000000005, "width": 182.61988830566406, "height": 25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b03", "roundness": null, "seed": 1557529373, "version": 34, "versionNonce": 540591763, "isDeleted": false, "boundElements": [], "updated": 1774385240033, "link": null, "locked": false, "text": "GPU Pool (worker)", "fontSize": 20, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "Xrb-UzhwTn1dzM7CWzaMv", "originalText": "GPU Pool (worker)", "autoResize": true, "lineHeight": 1.25 }, { "id": "Da3Q1Z9DLN5ZBRRvigDaG", "type": "arrow", "x": 1202.1354166666663, "y": 2889.2500000000005, "width": 264.59375, "height": 77.15625, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b04", "roundness": { "type": 2 }, "seed": 962439453, "version": 50, "versionNonce": 1509239155, "isDeleted": false, "boundElements": null, "updated": 1774385214371, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 264.59375, -77.15625 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "-l0uMlqcjcn8R0S3z00aM", "focus": 0.2201587027553122, "gap": 11.09375 }, "endBinding": { "elementId": "XrPldEc-JzLFQ1jwVb2N9", "focus": 0.535526449219992, "gap": 9.4375 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "oRNvZcpqJbmPmch_USP3s", "type": "arrow", "x": 1224.4791666666663, "y": 2916.6562500000005, "width": 250.125, "height": 73.15625, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b05", "roundness": { "type": 2 }, "seed": 1780002813, "version": 35, "versionNonce": 615291677, "isDeleted": false, "boundElements": null, "updated": 1774385216905, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 250.125, 73.15625 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "-l0uMlqcjcn8R0S3z00aM", "focus": -0.31702033092618453, "gap": 11.25 }, "endBinding": { "elementId": "CyQsFhm6jz_-lKxL_iLeV", "focus": -0.25565672867102907, "gap": 5.95703125 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "lNrUY4Em4xg7GhKPA1NWT", "type": "arrow", "x": 1752.5416666666663, "y": 2818.0937500000005, "width": 206.7265625, "height": 63.5390625, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b06", "roundness": { "type": 2 }, "seed": 1943634035, "version": 35, "versionNonce": 44125011, "isDeleted": false, "boundElements": null, "updated": 1774385219568, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 206.7265625, 63.5390625 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "XrPldEc-JzLFQ1jwVb2N9", "focus": -0.482830412382911, "gap": 8.0546875 }, "endBinding": { "elementId": "Xrb-UzhwTn1dzM7CWzaMv", "focus": -0.28658917870533274, "gap": 6.949218750000455 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "xUHuSXq_wec0CILYpRjnq", "type": "arrow", "x": 1752.3229166666663, "y": 3008.3750000000005, "width": 210.71875, "height": 111.6875, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b07", "roundness": { "type": 2 }, "seed": 1460376605, "version": 38, "versionNonce": 1727828147, "isDeleted": false, "boundElements": null, "updated": 1774385222179, "link": null, "locked": false, "points": [ [ 0, 0 ], [ 210.71875, -111.6875 ] ], "lastCommittedPoint": null, "startBinding": { "elementId": "CyQsFhm6jz_-lKxL_iLeV", "focus": 0.655863968449815, "gap": 15.35546875 }, "endBinding": { "elementId": "Xrb-UzhwTn1dzM7CWzaMv", "focus": 0.4654925693854583, "gap": 10.722656250000455 }, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false }, { "id": "N832BYA-KEHLojL-uz6FQ", "type": "text", "x": 1051.6666666666663, "y": 3240.0000000000005, "width": 603.219482421875, "height": 175, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b08", "roundness": null, "seed": 624262227, "version": 126, "versionNonce": 1470176115, "isDeleted": false, "boundElements": null, "updated": 1774385312752, "link": null, "locked": false, "text": "Redis Distributed Lock\n\nif redis.set('batch_lock',instance_id,nx=True,nx=True,ex=0.05)\n flush()\nelse\n\nSimple- no kafka", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Redis Distributed Lock\n\nif redis.set('batch_lock',instance_id,nx=True,nx=True,ex=0.05)\n flush()\nelse\n\nSimple- no kafka", "autoResize": true, "lineHeight": 1.25 }, { "id": "XAsz3SBg6klXk2Fzb1Rx0", "type": "text", "x": 1967.6666666666663, "y": 3212.0000000000005, "width": 367.2197570800781, "height": 125, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b09", "roundness": null, "seed": 539136915, "version": 96, "versionNonce": 457565907, "isDeleted": false, "boundElements": null, "updated": 1774385396303, "link": null, "locked": false, "text": "Scale up - fast trigger\nif gpu_utiliztion > 80% for 2 minutes\n -> provision_new_gpu()\nelif:\n", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Scale up - fast trigger\nif gpu_utiliztion > 80% for 2 minutes\n -> provision_new_gpu()\nelif:\n", "autoResize": true, "lineHeight": 1.25 }, { "id": "XJACoI6LHaUb1n3d6aHxc", "type": "text", "x": 1099.6666666666663, "y": 3524.0000000000005, "width": 476.03961181640625, "height": 50, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "b0A", "roundness": null, "seed": 1211000147, "version": 53, "versionNonce": 126161363, "isDeleted": false, "boundElements": null, "updated": 1774385420117, "link": null, "locked": false, "text": "Dynamic Rate Limiting - Green/Yellow/Red Zones\n", "fontSize": 20, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Dynamic Rate Limiting - Green/Yellow/Red Zones\n", "autoResize": true, "lineHeight": 1.25 } ], "appState": { "gridSize": 20, "gridStep": 5, "gridModeEnabled": false, "viewBackgroundColor": "#ffffff" }, "files": {} }