[{"data":1,"prerenderedAt":1165},["ShallowReactive",2],{"work-\u002Fwork\u002Fproject-dataharvester-etl-pipeline":3},{"id":4,"title":5,"body":6,"date":1148,"description":1149,"extension":1150,"externalUrl":1151,"featured":1152,"kind":1153,"meta":1154,"navigation":164,"path":1155,"seo":1156,"stem":1157,"tags":1158,"__hash__":1164},"work\u002Fwork\u002Fproject-dataharvester-etl-pipeline.md","Real estate data pipeline: MLS integration",{"type":7,"value":8,"toc":1136},"minimark",[9,14,19,23,32,35,39,42,58,61,65,70,98,102,109,324,327,338,341,348,558,561,572,579,793,796,800,832,836,843,846,849,856,893,896,903,906,913,1038,1041,1045,1111,1115,1118,1121,1124,1127,1132],[10,11,13],"h1",{"id":12},"building-a-scalable-real-estate-data-etl-pipeline","Building a scalable real estate data ETL pipeline",[15,16,18],"h2",{"id":17},"context","Context",[20,21,22],"p",{},"While working with a government agency client, I faced an interesting data engineering challenge. They needed to collect and analyze comprehensive data about real estate agencies and their properties from a proprietary Multiple Listing Service (MLS) system. The scope? Hundreds of thousands of property records that needed to be extracted, transformed, and analyzed while carefully respecting API rate limits and handling potential failures.",[24,25,26],"blockquote",{},[20,27,28],{},[29,30,31],"em",{},"Note: Out of respect for API terms and client confidentiality, I've anonymized the specific platform and client details. However, the technical challenge is common in enterprise data integration: working with rate-limited APIs, handling large datasets efficiently, and maintaining data integrity throughout the ETL process.",[20,33,34],{},"What made this particularly interesting was the balance between aggressive data collection and being a good API citizen – I needed to get this done efficiently but without overwhelming the source system.",[15,36,38],{"id":37},"what-i-built","What I Built",[20,40,41],{},"I developed a resilient ETL (Extract, Transform, Load) pipeline that could:",[43,44,45,49,52,55],"ol",{},[46,47,48],"li",{},"Extract property data from the MLS API in configurable batches",[46,50,51],{},"Process and deduplicate agency information on the fly",[46,53,54],{},"Transform the raw data into clean, analysis-ready CSV format",[46,56,57],{},"Handle interruptions gracefully with built-in checkpointing",[20,59,60],{},"The system was designed to be both robust and considerate – implementing smart rate limiting, token management, and data integrity checks throughout the pipeline.",[15,62,64],{"id":63},"technical-breakdown","Technical Breakdown",[66,67,69],"h3",{"id":68},"stack-tools","Stack & Tools",[71,72,73,80,86,92],"ul",{},[46,74,75,79],{},[76,77,78],"strong",{},"TypeScript\u002FNode.js",": Core implementation",[46,81,82,85],{},[76,83,84],{},"Axios",": HTTP client for API interactions",[46,87,88,91],{},[76,89,90],{},"File System (fs)",": For data persistence and checkpointing",[46,93,94,97],{},[76,95,96],{},"Path",": Cross-platform file path handling",[66,99,101],{"id":100},"key-architecture-decisions","Key Architecture Decisions",[43,103,104],{},[46,105,106],{},[76,107,108],{},"Chunked Data Processing",[110,111,116],"pre",{"className":112,"code":113,"language":114,"meta":115,"style":115},"language-typescript shiki shiki-themes material-theme-lighter github-light github-dark","const batchSize = 100;\nconst maxPropertiesPerFile = 70000;\n\nasync function fetchAllProperties() {\n  \u002F\u002F Create output directory if it doesn't exist\n  if (!fs.existsSync(outputDir)) {\n    fs.mkdirSync(outputDir);\n  }\n\n  \u002F\u002F Find the last file I was working on\n  let currentFileIndex = 0;\n  let allProperties: any[] = [];\n\n  \u002F\u002F ... chunked processing logic\n}\n","typescript","",[117,118,119,144,159,166,185,192,228,248,254,259,265,281,307,312,318],"code",{"__ignoreMap":115},[120,121,124,128,132,136,140],"span",{"class":122,"line":123},"line",1,[120,125,127],{"class":126},"sbsja","const",[120,129,131],{"class":130},"s_hVV"," batchSize",[120,133,135],{"class":134},"smGrS"," =",[120,137,139],{"class":138},"srdBf"," 100",[120,141,143],{"class":142},"sP7_E",";\n",[120,145,147,149,152,154,157],{"class":122,"line":146},2,[120,148,127],{"class":126},[120,150,151],{"class":130}," maxPropertiesPerFile",[120,153,135],{"class":134},[120,155,156],{"class":138}," 70000",[120,158,143],{"class":142},[120,160,162],{"class":122,"line":161},3,[120,163,165],{"emptyLinePlaceholder":164},true,"\n",[120,167,169,172,175,179,182],{"class":122,"line":168},4,[120,170,171],{"class":126},"async",[120,173,174],{"class":126}," function",[120,176,178],{"class":177},"sGLFI"," fetchAllProperties",[120,180,181],{"class":142},"()",[120,183,184],{"class":142}," {\n",[120,186,188],{"class":122,"line":187},5,[120,189,191],{"class":190},"sutJx","  \u002F\u002F Create output directory if it doesn't exist\n",[120,193,195,199,203,206,210,213,216,219,222,225],{"class":122,"line":194},6,[120,196,198],{"class":197},"sVHd0","  if",[120,200,202],{"class":201},"skxfh"," (",[120,204,205],{"class":134},"!",[120,207,209],{"class":208},"su5hD","fs",[120,211,212],{"class":142},".",[120,214,215],{"class":177},"existsSync",[120,217,218],{"class":201},"(",[120,220,221],{"class":208},"outputDir",[120,223,224],{"class":201},")) ",[120,226,227],{"class":142},"{\n",[120,229,231,234,236,239,241,243,246],{"class":122,"line":230},7,[120,232,233],{"class":208},"    fs",[120,235,212],{"class":142},[120,237,238],{"class":177},"mkdirSync",[120,240,218],{"class":201},[120,242,221],{"class":208},[120,244,245],{"class":201},")",[120,247,143],{"class":142},[120,249,251],{"class":122,"line":250},8,[120,252,253],{"class":142},"  }\n",[120,255,257],{"class":122,"line":256},9,[120,258,165],{"emptyLinePlaceholder":164},[120,260,262],{"class":122,"line":261},10,[120,263,264],{"class":190},"  \u002F\u002F Find the last file I was working on\n",[120,266,268,271,274,276,279],{"class":122,"line":267},11,[120,269,270],{"class":126},"  let",[120,272,273],{"class":208}," currentFileIndex",[120,275,135],{"class":134},[120,277,278],{"class":138}," 0",[120,280,143],{"class":142},[120,282,284,286,289,292,296,299,302,305],{"class":122,"line":283},12,[120,285,270],{"class":126},[120,287,288],{"class":208}," allProperties",[120,290,291],{"class":134},":",[120,293,295],{"class":294},"sZMiF"," any",[120,297,298],{"class":201},"[] ",[120,300,301],{"class":134},"=",[120,303,304],{"class":201}," []",[120,306,143],{"class":142},[120,308,310],{"class":122,"line":309},13,[120,311,165],{"emptyLinePlaceholder":164},[120,313,315],{"class":122,"line":314},14,[120,316,317],{"class":190},"  \u002F\u002F ... chunked processing logic\n",[120,319,321],{"class":122,"line":320},15,[120,322,323],{"class":142},"}\n",[20,325,326],{},"I implemented a chunked processing system that:",[71,328,329,332,335],{},[46,330,331],{},"Fetches data in small batches (100 properties)",[46,333,334],{},"Splits output into manageable files (~70K properties each)",[46,336,337],{},"Enables resume-ability if the process fails",[20,339,340],{},"This approach proved crucial when dealing with the full dataset of nearly 700,000 properties.",[43,342,343],{"start":146},[46,344,345],{},[76,346,347],{},"Resilient Error Handling",[110,349,351],{"className":112,"code":350,"language":114,"meta":115,"style":115},"try {\n  const response = await axios.post\u003CSearchResponse>(\n    API_ENDPOINT,\n    requestPayload,\n    { headers },\n  );\n} catch (error: any) {\n  if (error?.response?.status === 401) {\n    console.error(\n      \"Token expired! Please update the authorization token and run again\",\n    );\n    return;\n  }\n  \u002F\u002F For other errors, wait and retry\n  await delay(5000);\n  continue;\n}\n",[117,352,353,360,394,402,409,420,427,449,479,490,505,512,519,523,528,545,553],{"__ignoreMap":115},[120,354,355,358],{"class":122,"line":123},[120,356,357],{"class":197},"try",[120,359,184],{"class":142},[120,361,362,365,368,370,373,376,378,381,384,388,391],{"class":122,"line":146},[120,363,364],{"class":126},"  const",[120,366,367],{"class":130}," response",[120,369,135],{"class":134},[120,371,372],{"class":197}," await",[120,374,375],{"class":208}," axios",[120,377,212],{"class":142},[120,379,380],{"class":177},"post",[120,382,383],{"class":142},"\u003C",[120,385,387],{"class":386},"sbgvK","SearchResponse",[120,389,390],{"class":142},">",[120,392,393],{"class":201},"(\n",[120,395,396,399],{"class":122,"line":161},[120,397,398],{"class":130},"    API_ENDPOINT",[120,400,401],{"class":142},",\n",[120,403,404,407],{"class":122,"line":168},[120,405,406],{"class":208},"    requestPayload",[120,408,401],{"class":142},[120,410,411,414,417],{"class":122,"line":187},[120,412,413],{"class":142},"    {",[120,415,416],{"class":208}," headers",[120,418,419],{"class":142}," },\n",[120,421,422,425],{"class":122,"line":194},[120,423,424],{"class":201},"  )",[120,426,143],{"class":142},[120,428,429,432,435,437,441,443,445,447],{"class":122,"line":230},[120,430,431],{"class":142},"}",[120,433,434],{"class":197}," catch",[120,436,202],{"class":142},[120,438,440],{"class":439},"s99_P","error",[120,442,291],{"class":134},[120,444,295],{"class":294},[120,446,245],{"class":142},[120,448,184],{"class":142},[120,450,451,453,455,457,460,463,465,468,471,474,477],{"class":122,"line":250},[120,452,198],{"class":197},[120,454,202],{"class":201},[120,456,440],{"class":208},[120,458,459],{"class":142},"?.",[120,461,462],{"class":208},"response",[120,464,459],{"class":142},[120,466,467],{"class":208},"status",[120,469,470],{"class":134}," ===",[120,472,473],{"class":138}," 401",[120,475,476],{"class":201},") ",[120,478,227],{"class":142},[120,480,481,484,486,488],{"class":122,"line":256},[120,482,483],{"class":208},"    console",[120,485,212],{"class":142},[120,487,440],{"class":177},[120,489,393],{"class":201},[120,491,492,496,500,503],{"class":122,"line":261},[120,493,495],{"class":494},"sjJ54","      \"",[120,497,499],{"class":498},"s_sjI","Token expired! Please update the authorization token and run again",[120,501,502],{"class":494},"\"",[120,504,401],{"class":142},[120,506,507,510],{"class":122,"line":267},[120,508,509],{"class":201},"    )",[120,511,143],{"class":142},[120,513,514,517],{"class":122,"line":283},[120,515,516],{"class":197},"    return",[120,518,143],{"class":142},[120,520,521],{"class":122,"line":309},[120,522,253],{"class":142},[120,524,525],{"class":122,"line":314},[120,526,527],{"class":190},"  \u002F\u002F For other errors, wait and retry\n",[120,529,530,533,536,538,541,543],{"class":122,"line":320},[120,531,532],{"class":197},"  await",[120,534,535],{"class":177}," delay",[120,537,218],{"class":201},[120,539,540],{"class":138},"5000",[120,542,245],{"class":201},[120,544,143],{"class":142},[120,546,548,551],{"class":122,"line":547},16,[120,549,550],{"class":197},"  continue",[120,552,143],{"class":142},[120,554,556],{"class":122,"line":555},17,[120,557,323],{"class":142},[20,559,560],{},"The system handles:",[71,562,563,566,569],{},[46,564,565],{},"Token expiration gracefully",[46,567,568],{},"Network failures with automatic retries",[46,570,571],{},"Rate limiting through intelligent delays",[43,573,574],{"start":161},[46,575,576],{},[76,577,578],{},"Smart Deduplication",[110,580,582],{"className":112,"code":581,"language":114,"meta":115,"style":115},"const agencyMap = new Map\u003Cstring, Agency>();\n\n\u002F\u002F During property processing\nproperties.forEach((property) => {\n  const { agency } = property;\n  if (agency?.email) {\n    agencyMap.set(agency.email, {\n      email: agency.email,\n      name: agency.name,\n      phone: agency.phone,\n      websiteUrl: agency.websiteUrl,\n    });\n  }\n});\n",[117,583,584,616,620,625,649,669,687,709,724,740,756,772,781,785],{"__ignoreMap":115},[120,585,586,588,591,593,596,599,601,604,607,610,612,614],{"class":122,"line":123},[120,587,127],{"class":126},[120,589,590],{"class":130}," agencyMap",[120,592,135],{"class":134},[120,594,595],{"class":134}," new",[120,597,598],{"class":177}," Map",[120,600,383],{"class":142},[120,602,603],{"class":294},"string",[120,605,606],{"class":142},",",[120,608,609],{"class":386}," Agency",[120,611,390],{"class":142},[120,613,181],{"class":208},[120,615,143],{"class":142},[120,617,618],{"class":122,"line":146},[120,619,165],{"emptyLinePlaceholder":164},[120,621,622],{"class":122,"line":161},[120,623,624],{"class":190},"\u002F\u002F During property processing\n",[120,626,627,630,632,635,637,639,642,644,647],{"class":122,"line":168},[120,628,629],{"class":208},"properties",[120,631,212],{"class":142},[120,633,634],{"class":177},"forEach",[120,636,218],{"class":208},[120,638,218],{"class":142},[120,640,641],{"class":439},"property",[120,643,245],{"class":142},[120,645,646],{"class":126}," =>",[120,648,184],{"class":142},[120,650,651,653,656,659,662,664,667],{"class":122,"line":187},[120,652,364],{"class":126},[120,654,655],{"class":142}," {",[120,657,658],{"class":130}," agency",[120,660,661],{"class":142}," }",[120,663,135],{"class":134},[120,665,666],{"class":208}," property",[120,668,143],{"class":142},[120,670,671,673,675,678,680,683,685],{"class":122,"line":194},[120,672,198],{"class":197},[120,674,202],{"class":201},[120,676,677],{"class":208},"agency",[120,679,459],{"class":142},[120,681,682],{"class":208},"email",[120,684,476],{"class":201},[120,686,227],{"class":142},[120,688,689,692,694,697,699,701,703,705,707],{"class":122,"line":230},[120,690,691],{"class":208},"    agencyMap",[120,693,212],{"class":142},[120,695,696],{"class":177},"set",[120,698,218],{"class":201},[120,700,677],{"class":208},[120,702,212],{"class":142},[120,704,682],{"class":208},[120,706,606],{"class":142},[120,708,184],{"class":142},[120,710,711,714,716,718,720,722],{"class":122,"line":250},[120,712,713],{"class":201},"      email",[120,715,291],{"class":142},[120,717,658],{"class":208},[120,719,212],{"class":142},[120,721,682],{"class":208},[120,723,401],{"class":142},[120,725,726,729,731,733,735,738],{"class":122,"line":256},[120,727,728],{"class":201},"      name",[120,730,291],{"class":142},[120,732,658],{"class":208},[120,734,212],{"class":142},[120,736,737],{"class":208},"name",[120,739,401],{"class":142},[120,741,742,745,747,749,751,754],{"class":122,"line":261},[120,743,744],{"class":201},"      phone",[120,746,291],{"class":142},[120,748,658],{"class":208},[120,750,212],{"class":142},[120,752,753],{"class":208},"phone",[120,755,401],{"class":142},[120,757,758,761,763,765,767,770],{"class":122,"line":267},[120,759,760],{"class":201},"      websiteUrl",[120,762,291],{"class":142},[120,764,658],{"class":208},[120,766,212],{"class":142},[120,768,769],{"class":208},"websiteUrl",[120,771,401],{"class":142},[120,773,774,777,779],{"class":122,"line":283},[120,775,776],{"class":142},"    }",[120,778,245],{"class":201},[120,780,143],{"class":142},[120,782,783],{"class":122,"line":309},[120,784,253],{"class":142},[120,786,787,789,791],{"class":122,"line":314},[120,788,431],{"class":142},[120,790,245],{"class":208},[120,792,143],{"class":142},[20,794,795],{},"Used email as a unique key to deduplicate agency information across properties, ensuring data consistency while minimizing memory usage.",[66,797,799],{"id":798},"edge-cases-handled","Edge Cases Handled",[71,801,802,808,814,820,826],{},[46,803,804,807],{},[76,805,806],{},"Partial File Completion",": The system tracks progress and can resume from the last successful batch",[46,809,810,813],{},[76,811,812],{},"API Token Management",": Graceful handling of token expiration with clear error messages",[46,815,816,819],{},[76,817,818],{},"Data Quality",": Handling of missing or malformed data without breaking the pipeline",[46,821,822,825],{},[76,823,824],{},"Process Interruption",": File-based checkpointing enables resume-ability",[46,827,828,831],{},[76,829,830],{},"Memory Management",": Streaming approach to handle large datasets efficiently",[15,833,835],{"id":834},"what-i-learned","What I Learned",[43,837,838],{},[46,839,840],{},[76,841,842],{},"Batch Processing Trade-offs",[20,844,845],{},"Initially, I tried processing all data in memory. This worked fine during testing with small datasets but quickly became problematic when dealing with the full dataset. Breaking the data into chunks with file-based checkpointing proved more reliable, though slightly slower.",[20,847,848],{},"The key insight? Sometimes trading raw speed for reliability is the right choice, especially when dealing with large-scale data extraction.",[43,850,851],{"start":146},[46,852,853],{},[76,854,855],{},"Rate Limiting Strategy",[110,857,859],{"className":112,"code":858,"language":114,"meta":115,"style":115},"await delay(Math.random() * 1000); \u002F\u002F Random delay between 0-1 seconds\n",[117,860,861],{"__ignoreMap":115},[120,862,863,866,868,871,873,876,879,882,885,887,890],{"class":122,"line":123},[120,864,865],{"class":197},"await",[120,867,535],{"class":177},[120,869,870],{"class":208},"(Math",[120,872,212],{"class":142},[120,874,875],{"class":177},"random",[120,877,878],{"class":208},"() ",[120,880,881],{"class":134},"*",[120,883,884],{"class":138}," 1000",[120,886,245],{"class":208},[120,888,889],{"class":142},";",[120,891,892],{"class":190}," \u002F\u002F Random delay between 0-1 seconds\n",[20,894,895],{},"Instead of fixed delays, implementing random delays between requests helped avoid predictable patterns that might trigger API defenses. This simple change made our script behave more like natural traffic.",[43,897,898],{"start":161},[46,899,900],{},[76,901,902],{},"Data Integrity > Speed",[20,904,905],{},"Using a Map for deduplication was more efficient than array-based filtering, especially when dealing with tens of thousands of records. The slight memory overhead was worth it for the guaranteed uniqueness and O(1) lookup times.",[43,907,908],{"start":168},[46,909,910],{},[76,911,912],{},"Progress Visibility",[110,914,916],{"className":112,"code":915,"language":114,"meta":115,"style":115},"console.log(\n  `Saved ${skip + allProperties.length}\u002F${total} properties (${(\n    ((skip + allProperties.length) \u002F total) *\n    100\n  ).toFixed(2)}%)`,\n);\n",[117,917,918,930,974,1001,1006,1032],{"__ignoreMap":115},[120,919,920,923,925,928],{"class":122,"line":123},[120,921,922],{"class":208},"console",[120,924,212],{"class":142},[120,926,927],{"class":177},"log",[120,929,393],{"class":208},[120,931,932,935,938,941,944,947,949,951,954,956,959,961,964,966,969,971],{"class":122,"line":146},[120,933,934],{"class":494},"  `",[120,936,937],{"class":498},"Saved ",[120,939,940],{"class":494},"${",[120,942,943],{"class":208},"skip",[120,945,946],{"class":134}," +",[120,948,288],{"class":208},[120,950,212],{"class":494},[120,952,953],{"class":130},"length",[120,955,431],{"class":494},[120,957,958],{"class":498},"\u002F",[120,960,940],{"class":494},[120,962,963],{"class":208},"total",[120,965,431],{"class":494},[120,967,968],{"class":498}," properties (",[120,970,940],{"class":494},[120,972,393],{"class":973},"sfo-9",[120,975,976,979,981,983,985,987,989,991,993,996,998],{"class":122,"line":161},[120,977,978],{"class":973},"    ((",[120,980,943],{"class":208},[120,982,946],{"class":134},[120,984,288],{"class":208},[120,986,212],{"class":494},[120,988,953],{"class":130},[120,990,476],{"class":973},[120,992,958],{"class":134},[120,994,995],{"class":208}," total",[120,997,476],{"class":973},[120,999,1000],{"class":134},"*\n",[120,1002,1003],{"class":122,"line":168},[120,1004,1005],{"class":138},"    100\n",[120,1007,1008,1010,1012,1015,1017,1020,1022,1024,1027,1030],{"class":122,"line":187},[120,1009,424],{"class":973},[120,1011,212],{"class":494},[120,1013,1014],{"class":177},"toFixed",[120,1016,218],{"class":973},[120,1018,1019],{"class":138},"2",[120,1021,245],{"class":973},[120,1023,431],{"class":494},[120,1025,1026],{"class":498},"%)",[120,1028,1029],{"class":494},"`",[120,1031,401],{"class":142},[120,1033,1034,1036],{"class":122,"line":194},[120,1035,245],{"class":208},[120,1037,143],{"class":142},[20,1039,1040],{},"Adding detailed progress logging helped track long-running processes and identify bottlenecks. This became invaluable when the client asked for status updates or when debugging issues.",[15,1042,1044],{"id":1043},"whats-next","What's Next?",[43,1046,1047,1063,1079,1095],{},[46,1048,1049,1052],{},[76,1050,1051],{},"Performance Optimization",[71,1053,1054,1057,1060],{},[46,1055,1056],{},"Implement parallel processing with worker threads",[46,1058,1059],{},"Investigate streaming CSV generation for lower memory usage",[46,1061,1062],{},"Add batch size auto-tuning based on API response times",[46,1064,1065,1068],{},[76,1066,1067],{},"Data Validation",[71,1069,1070,1073,1076],{},[46,1071,1072],{},"Add JSON schema validation for API responses",[46,1074,1075],{},"Implement data quality scoring",[46,1077,1078],{},"Add automated anomaly detection",[46,1080,1081,1084],{},[76,1082,1083],{},"Monitoring & Observability",[71,1085,1086,1089,1092],{},[46,1087,1088],{},"Add proper metrics collection",[46,1090,1091],{},"Implement real-time progress tracking",[46,1093,1094],{},"Create a dashboard for pipeline status",[46,1096,1097,1100],{},[76,1098,1099],{},"Configuration Management",[71,1101,1102,1105,1108],{},[46,1103,1104],{},"Move hardcoded values to configuration files",[46,1106,1107],{},"Add environment-specific settings",[46,1109,1110],{},"Implement feature flags for different processing modes",[15,1112,1114],{"id":1113},"key-takeaways","Key Takeaways",[20,1116,1117],{},"In this case I learned importance of building data pipelines that are not just functional, but also resilient and maintainable. The extra time spent on error handling and progress tracking paid off many times over during the actual data collection phase.",[20,1119,1120],{},"It definitely highlighted how technical challenges often require balancing competing concerns – in this case, speed vs. reliability, and thoroughness vs. API courtesy.",[20,1122,1123],{},"The end result was a robust system that successfully processed nearly 700,000 property records, extracting valuable insights for our client while maintaining data integrity and system reliability throughout the process.",[1125,1126],"hr",{},[20,1128,1129],{},[29,1130,1131],{},"Note: This case study has been anonymized to protect client confidentiality while preserving the technical insights and learning opportunities from the project.",[1133,1134,1135],"style",{},"html pre.shiki code .sbsja, html code.shiki .sbsja{--shiki-light:#9C3EDA;--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .s_hVV, html code.shiki .s_hVV{--shiki-light:#90A4AE;--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .smGrS, html code.shiki .smGrS{--shiki-light:#39ADB5;--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .srdBf, html code.shiki .srdBf{--shiki-light:#F76D47;--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sP7_E, html code.shiki .sP7_E{--shiki-light:#39ADB5;--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sGLFI, html code.shiki .sGLFI{--shiki-light:#6182B8;--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sutJx, html code.shiki .sutJx{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#6A737D;--shiki-default-font-style:inherit;--shiki-dark:#6A737D;--shiki-dark-font-style:inherit}html pre.shiki code .sVHd0, html code.shiki .sVHd0{--shiki-light:#39ADB5;--shiki-light-font-style:italic;--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit}html pre.shiki code .skxfh, html code.shiki .skxfh{--shiki-light:#E53935;--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .su5hD, html code.shiki .su5hD{--shiki-light:#90A4AE;--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sZMiF, html code.shiki .sZMiF{--shiki-light:#E2931D;--shiki-default:#005CC5;--shiki-dark:#79B8FF}html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sbgvK, html code.shiki .sbgvK{--shiki-light:#E2931D;--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .s99_P, html code.shiki .s99_P{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#E36209;--shiki-default-font-style:inherit;--shiki-dark:#FFAB70;--shiki-dark-font-style:inherit}html pre.shiki code .sjJ54, html code.shiki .sjJ54{--shiki-light:#39ADB5;--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .s_sjI, html code.shiki .s_sjI{--shiki-light:#91B859;--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .sfo-9, html code.shiki .sfo-9{--shiki-light:#90A4AE;--shiki-default:#032F62;--shiki-dark:#9ECBFF}",{"title":115,"searchDepth":146,"depth":146,"links":1137},[1138,1139,1140,1145,1146,1147],{"id":17,"depth":146,"text":18},{"id":37,"depth":146,"text":38},{"id":63,"depth":146,"text":64,"children":1141},[1142,1143,1144],{"id":68,"depth":161,"text":69},{"id":100,"depth":161,"text":101},{"id":798,"depth":161,"text":799},{"id":834,"depth":146,"text":835},{"id":1043,"depth":146,"text":1044},{"id":1113,"depth":146,"text":1114},"2025-02-26","Developed a robust ETL pipeline to extract, transform, and analyze real estate agency data from a proprietary MLS system, handling rate limits and large datasets efficiently.","md",null,false,"case-study",{},"\u002Fwork\u002Fproject-dataharvester-etl-pipeline",{"title":5,"description":1149},"work\u002Fproject-dataharvester-etl-pipeline",[1159,1160,1161,1162,1163],"TypeScript","Node.js","ETL","Data Processing","REST API","tbmSJYNZnNxlfYQ8uu37AUNO-vY6-1aVEt6kd8Pawgk",1780955296664]