{"id":562,"date":"2020-07-29T12:00:00","date_gmt":"2020-07-29T16:00:00","guid":{"rendered":"https:\/\/www.pwvconsultants.com\/blog\/?p=562"},"modified":"2020-07-27T18:49:27","modified_gmt":"2020-07-27T22:49:27","slug":"time-series-and-flat-file-databases","status":"publish","type":"post","link":"https:\/\/www.pwvconsultants.com\/blog\/time-series-and-flat-file-databases\/","title":{"rendered":"Time Series and Flat File Databases"},"content":{"rendered":"\n<p>Time series and flat file storage solutions are both specialized databases. Like <a href=\"http:\/\/log-stores-are-specialized-databases\" target=\"_blank\" aria-label=\"undefined (opens in a new tab)\" rel=\"noreferrer noopener\">log stores<\/a>, they were designed with a specific purpose and attempting to do much outside the realm of what they are good at will result in inefficiency and hang ups. In this installment of our <a aria-label=\"undefined (opens in a new tab)\" href=\"https:\/\/www.pwvconsultants.com\/blog\/datastores-an-overview-of-common-types\/\" target=\"_blank\" rel=\"noreferrer noopener\">series on datastores<\/a>, let\u2019s discuss what time series and flat file databases are and when to use them.<\/p>\n\n\n\n<p>Time series datastores are usually intended for extremely fast-moving data. Millions, billions of data points per minute or even per second. But it\u2019s also chronological and time stamped. It\u2019s not meant to be schematic data. It\u2019s not meant to do complex queries or relate data to other data. It is meant to take tons and tons of data in near real time, so you are getting data in and time stamped appropriately. You\u2019re not waiting and queuing data while complex writes occur or a table is locked on a complicated join. The function of time series data stores is limited and when you try to do things beyond its limitations, it\u2019s going to put up an index finger and tell you to hang on for a minute. Only that one minute is closer to 30 minutes, or sometimes longer<\/p>\n\n\n\n<p>Time series stores are not efficient for anything other than high throughput, rapid firing data capture. And it\u2019s usually meant for simplistic data, but because it handles a massive amount of transactions, it can still be expensive. The data it handles is typically something that happens in the memory of a large server. And while data is written down later to back itself up, there is still a rate of error that must be accepted. And maybe there are circumstances where you care less about the time series itself and just need to capture massive amounts of data in a short window, time series databases can help. Just remember that its function is limited and doesn\u2019t allow for highly relatable data and cross-sectioning and complex queries.<\/p>\n\n\n\n<p>Flat file storage can be anything from storing items in your hard drive and flat files in various formats to S3 buckets or something similar. It\u2019s simply for storage. You\u2019re writing it to a file for later use. You might put it in a JSON file or several other formats, but you have the data and it\u2019s easily parsed and consumed. This is a form of data management that is often overlooked.<\/p>\n\n\n\n<p>There is latency built into using flat file storage that\u2019s native. If you are going to write to a general database, oftentimes what is happening is you\u2019re working in the RAM on a computer. Later, that data will get put onto the actual hard drive, especially if you\u2019re taking the database and optimizing  it to make it work fast.<\/p>\n\n\n\n<p>With flat file storage, what you\u2019re doing is writing to files instead of writing to memory or writing to a datastore. This is how you have data that\u2019s not ephemeral and won\u2019t disappear in a traditional database. But sometimes it\u2019s easier to do it directly, on your own, and sometimes it\u2019s faster, depending on what you are doing and how you\u2019re interacting with the data itself. Because of how it works, it\u2019s also a method to back up data and it\u2019s almost always cheaper than other methods of data storage.<\/p>\n\n\n\n<p>Time series databases and flat file storage are like other datastores in that they are efficient at what they were designed for. They are both specialized for a specific purpose and are limited to that purpose, so trying to use them for something else makes them inefficient and will cause issues. It is usually a good idea to employ more than one form of data storage to enable various methods of working with your data. Understanding how each form of storage works is the first step to determining which storage method your business would benefit from the most.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Time series and flat file databases are specialized for specific use. You must stick to that appropriate use or efficiency fails.<\/p>\n","protected":false},"author":1,"featured_media":585,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[961,958],"tags":[925,1391,1273,1272,1278,1390,1285],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v22.2 (Yoast SEO v22.2) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Time Series and Flat File Databases - PWV Consultants<\/title>\n<meta name=\"description\" content=\"Time series and flat file databases are specialized for specific use. You must stick to that appropriate use or efficiency fails.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.pwvconsultants.com\/blog\/time-series-and-flat-file-databases\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Time Series and Flat File Databases\" \/>\n<meta property=\"og:description\" content=\"Time series and flat file databases are specialized for specific use. You must stick to that appropriate use or efficiency fails.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.pwvconsultants.com\/blog\/time-series-and-flat-file-databases\/\" \/>\n<meta property=\"og:site_name\" content=\"PWV Consultants\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/PWV-Consultants-110444033947964\" \/>\n<meta property=\"article:published_time\" content=\"2020-07-29T16:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2020-07-27T22:49:27+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.pwvconsultants.com\/blog\/wp-content\/uploads\/2020\/07\/Time-and-Log-Stores-Article-scaled.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"2560\" \/>\n\t<meta property=\"og:image:height\" content=\"1920\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Pieter VanIperen\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@PWV_Consultants\" \/>\n<meta name=\"twitter:site\" content=\"@PWV_Consultants\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Pieter VanIperen\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/time-series-and-flat-file-databases\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/time-series-and-flat-file-databases\/\"},\"author\":{\"name\":\"Pieter VanIperen\",\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/#\/schema\/person\/c15d5d40126a8ad906cb3067de95f8d4\"},\"headline\":\"Time Series and Flat File Databases\",\"datePublished\":\"2020-07-29T16:00:00+00:00\",\"dateModified\":\"2020-07-27T22:49:27+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/time-series-and-flat-file-databases\/\"},\"wordCount\":676,\"publisher\":{\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/time-series-and-flat-file-databases\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.pwvconsultants.com\/blog\/wp-content\/uploads\/2020\/07\/Time-and-Log-Stores-Article-scaled.jpg\",\"keywords\":[\"data\",\"data management\",\"Database\",\"Datastore\",\"Flat File\",\"Information Technology\",\"Time Series\"],\"articleSection\":[\"Data Stores\",\"Information Technology\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/time-series-and-flat-file-databases\/\",\"url\":\"https:\/\/www.pwvconsultants.com\/blog\/time-series-and-flat-file-databases\/\",\"name\":\"Time Series and Flat File Databases - PWV Consultants\",\"isPartOf\":{\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/time-series-and-flat-file-databases\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/time-series-and-flat-file-databases\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.pwvconsultants.com\/blog\/wp-content\/uploads\/2020\/07\/Time-and-Log-Stores-Article-scaled.jpg\",\"datePublished\":\"2020-07-29T16:00:00+00:00\",\"dateModified\":\"2020-07-27T22:49:27+00:00\",\"description\":\"Time series and flat file databases are specialized for specific use. You must stick to that appropriate use or efficiency fails.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/time-series-and-flat-file-databases\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.pwvconsultants.com\/blog\/time-series-and-flat-file-databases\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/time-series-and-flat-file-databases\/#primaryimage\",\"url\":\"https:\/\/www.pwvconsultants.com\/blog\/wp-content\/uploads\/2020\/07\/Time-and-Log-Stores-Article-scaled.jpg\",\"contentUrl\":\"https:\/\/www.pwvconsultants.com\/blog\/wp-content\/uploads\/2020\/07\/Time-and-Log-Stores-Article-scaled.jpg\",\"width\":2560,\"height\":1920},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/time-series-and-flat-file-databases\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.pwvconsultants.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Time Series and Flat File Databases\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/#website\",\"url\":\"https:\/\/www.pwvconsultants.com\/blog\/\",\"name\":\"PWV Consultants\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.pwvconsultants.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/#organization\",\"name\":\"PWV Consultants\",\"url\":\"https:\/\/www.pwvconsultants.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.pwvconsultants.com\/blog\/wp-content\/uploads\/2020\/04\/logo-alternate-e1585773530392.png\",\"contentUrl\":\"https:\/\/www.pwvconsultants.com\/blog\/wp-content\/uploads\/2020\/04\/logo-alternate-e1585773530392.png\",\"width\":98,\"height\":84,\"caption\":\"PWV Consultants\"},\"image\":{\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/PWV-Consultants-110444033947964\",\"https:\/\/twitter.com\/PWV_Consultants\",\"https:\/\/www.linkedin.com\/company\/pwv-consultants\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/#\/schema\/person\/c15d5d40126a8ad906cb3067de95f8d4\",\"name\":\"Pieter VanIperen\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/8b294918257a810803e2befc9a71b7bc?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/8b294918257a810803e2befc9a71b7bc?s=96&d=mm&r=g\",\"caption\":\"Pieter VanIperen\"},\"description\":\"PWV Consultants is a boutique group of industry leaders and influencers from the digital tech, security and design industries that acts as trusted technical partners for many Fortune 500 companies, high-visibility startups, universities, defense agencies, and NGOs. Founded by 20-year software engineering veterans, who have founded or co-founder several companies. PWV experts act as a trusted advisors and mentors to numerous early stage startups, and have held the titles of software and software security executive, consultant and professor. PWV's expert consulting and advisory work spans several high impact industries in finance, media, medical tech, and defense contracting. PWV's founding experts also authored the highly influential precursor HAZL (jADE) programming language.\",\"sameAs\":[\"https:\/\/www.linkedin.com\/company\/pwv-consultants\"]}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Time Series and Flat File Databases - PWV Consultants","description":"Time series and flat file databases are specialized for specific use. You must stick to that appropriate use or efficiency fails.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.pwvconsultants.com\/blog\/time-series-and-flat-file-databases\/","og_locale":"en_US","og_type":"article","og_title":"Time Series and Flat File Databases","og_description":"Time series and flat file databases are specialized for specific use. You must stick to that appropriate use or efficiency fails.","og_url":"https:\/\/www.pwvconsultants.com\/blog\/time-series-and-flat-file-databases\/","og_site_name":"PWV Consultants","article_publisher":"https:\/\/www.facebook.com\/PWV-Consultants-110444033947964","article_published_time":"2020-07-29T16:00:00+00:00","article_modified_time":"2020-07-27T22:49:27+00:00","og_image":[{"width":2560,"height":1920,"url":"https:\/\/www.pwvconsultants.com\/blog\/wp-content\/uploads\/2020\/07\/Time-and-Log-Stores-Article-scaled.jpg","type":"image\/jpeg"}],"author":"Pieter VanIperen","twitter_card":"summary_large_image","twitter_creator":"@PWV_Consultants","twitter_site":"@PWV_Consultants","twitter_misc":{"Written by":"Pieter VanIperen","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.pwvconsultants.com\/blog\/time-series-and-flat-file-databases\/#article","isPartOf":{"@id":"https:\/\/www.pwvconsultants.com\/blog\/time-series-and-flat-file-databases\/"},"author":{"name":"Pieter VanIperen","@id":"https:\/\/www.pwvconsultants.com\/blog\/#\/schema\/person\/c15d5d40126a8ad906cb3067de95f8d4"},"headline":"Time Series and Flat File Databases","datePublished":"2020-07-29T16:00:00+00:00","dateModified":"2020-07-27T22:49:27+00:00","mainEntityOfPage":{"@id":"https:\/\/www.pwvconsultants.com\/blog\/time-series-and-flat-file-databases\/"},"wordCount":676,"publisher":{"@id":"https:\/\/www.pwvconsultants.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.pwvconsultants.com\/blog\/time-series-and-flat-file-databases\/#primaryimage"},"thumbnailUrl":"https:\/\/www.pwvconsultants.com\/blog\/wp-content\/uploads\/2020\/07\/Time-and-Log-Stores-Article-scaled.jpg","keywords":["data","data management","Database","Datastore","Flat File","Information Technology","Time Series"],"articleSection":["Data Stores","Information Technology"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.pwvconsultants.com\/blog\/time-series-and-flat-file-databases\/","url":"https:\/\/www.pwvconsultants.com\/blog\/time-series-and-flat-file-databases\/","name":"Time Series and Flat File Databases - PWV Consultants","isPartOf":{"@id":"https:\/\/www.pwvconsultants.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.pwvconsultants.com\/blog\/time-series-and-flat-file-databases\/#primaryimage"},"image":{"@id":"https:\/\/www.pwvconsultants.com\/blog\/time-series-and-flat-file-databases\/#primaryimage"},"thumbnailUrl":"https:\/\/www.pwvconsultants.com\/blog\/wp-content\/uploads\/2020\/07\/Time-and-Log-Stores-Article-scaled.jpg","datePublished":"2020-07-29T16:00:00+00:00","dateModified":"2020-07-27T22:49:27+00:00","description":"Time series and flat file databases are specialized for specific use. You must stick to that appropriate use or efficiency fails.","breadcrumb":{"@id":"https:\/\/www.pwvconsultants.com\/blog\/time-series-and-flat-file-databases\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.pwvconsultants.com\/blog\/time-series-and-flat-file-databases\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.pwvconsultants.com\/blog\/time-series-and-flat-file-databases\/#primaryimage","url":"https:\/\/www.pwvconsultants.com\/blog\/wp-content\/uploads\/2020\/07\/Time-and-Log-Stores-Article-scaled.jpg","contentUrl":"https:\/\/www.pwvconsultants.com\/blog\/wp-content\/uploads\/2020\/07\/Time-and-Log-Stores-Article-scaled.jpg","width":2560,"height":1920},{"@type":"BreadcrumbList","@id":"https:\/\/www.pwvconsultants.com\/blog\/time-series-and-flat-file-databases\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.pwvconsultants.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Time Series and Flat File Databases"}]},{"@type":"WebSite","@id":"https:\/\/www.pwvconsultants.com\/blog\/#website","url":"https:\/\/www.pwvconsultants.com\/blog\/","name":"PWV Consultants","description":"","publisher":{"@id":"https:\/\/www.pwvconsultants.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.pwvconsultants.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.pwvconsultants.com\/blog\/#organization","name":"PWV Consultants","url":"https:\/\/www.pwvconsultants.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.pwvconsultants.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.pwvconsultants.com\/blog\/wp-content\/uploads\/2020\/04\/logo-alternate-e1585773530392.png","contentUrl":"https:\/\/www.pwvconsultants.com\/blog\/wp-content\/uploads\/2020\/04\/logo-alternate-e1585773530392.png","width":98,"height":84,"caption":"PWV Consultants"},"image":{"@id":"https:\/\/www.pwvconsultants.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/PWV-Consultants-110444033947964","https:\/\/twitter.com\/PWV_Consultants","https:\/\/www.linkedin.com\/company\/pwv-consultants"]},{"@type":"Person","@id":"https:\/\/www.pwvconsultants.com\/blog\/#\/schema\/person\/c15d5d40126a8ad906cb3067de95f8d4","name":"Pieter VanIperen","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.pwvconsultants.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/8b294918257a810803e2befc9a71b7bc?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/8b294918257a810803e2befc9a71b7bc?s=96&d=mm&r=g","caption":"Pieter VanIperen"},"description":"PWV Consultants is a boutique group of industry leaders and influencers from the digital tech, security and design industries that acts as trusted technical partners for many Fortune 500 companies, high-visibility startups, universities, defense agencies, and NGOs. Founded by 20-year software engineering veterans, who have founded or co-founder several companies. PWV experts act as a trusted advisors and mentors to numerous early stage startups, and have held the titles of software and software security executive, consultant and professor. PWV's expert consulting and advisory work spans several high impact industries in finance, media, medical tech, and defense contracting. PWV's founding experts also authored the highly influential precursor HAZL (jADE) programming language.","sameAs":["https:\/\/www.linkedin.com\/company\/pwv-consultants"]}]}},"_links":{"self":[{"href":"https:\/\/www.pwvconsultants.com\/blog\/wp-json\/wp\/v2\/posts\/562"}],"collection":[{"href":"https:\/\/www.pwvconsultants.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.pwvconsultants.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.pwvconsultants.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.pwvconsultants.com\/blog\/wp-json\/wp\/v2\/comments?post=562"}],"version-history":[{"count":5,"href":"https:\/\/www.pwvconsultants.com\/blog\/wp-json\/wp\/v2\/posts\/562\/revisions"}],"predecessor-version":[{"id":590,"href":"https:\/\/www.pwvconsultants.com\/blog\/wp-json\/wp\/v2\/posts\/562\/revisions\/590"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.pwvconsultants.com\/blog\/wp-json\/wp\/v2\/media\/585"}],"wp:attachment":[{"href":"https:\/\/www.pwvconsultants.com\/blog\/wp-json\/wp\/v2\/media?parent=562"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.pwvconsultants.com\/blog\/wp-json\/wp\/v2\/categories?post=562"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.pwvconsultants.com\/blog\/wp-json\/wp\/v2\/tags?post=562"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}