· Gary O'Keeffe · AI & Data · 5 min read
Where Dark Data Hides in a Lab
Most labs have more dark data than they realize. Here's where to look and how to find it in a week.
In our experience, most labs have dark data. It’s the information you’ve collected but don’t use—not because it’s worthless, but because you can’t find it or don’t know it exists.
Many teams underestimate how much dark data they have. They know about the obvious stuff: the network drive with years of project folders, the retired instrument PC in the corner. They often don’t know about the less obvious stuff, which can be more valuable.
The usual suspects
Start with the places most teams know about but haven’t inventoried.
Network drives. Most labs have at least one. They accumulate project folders, personal folders, “archive” folders, and folders with names like “DO NOT DELETE - OLD.” The data is technically accessible, but often nobody knows what’s in there. The person who organized it may have left years ago.
Instrument PCs. Many instruments write data to local drives. The data is supposed to be copied somewhere, but that process often depends on someone remembering to do it. In teams we’ve worked with, we’ve found instrument PCs with years of data that exists nowhere else.
Laptops. Scientists often store working data on their laptops. It’s convenient during a project and forgotten after. When someone leaves, their laptop may get wiped with data that was never backed up.
Email. Results get sent as attachments. Protocols get shared in threads. Decisions get made in replies. None of it is easily searchable or linked to anything else.
These are the usual suspects. A quick survey will turn them up. But they’re not where the most valuable dark data hides.
The less obvious places
LIMS and ELN metadata. Your LIMS tracks samples and experiments. Your ELN records what was done. But the raw data—the instrument outputs, the images, the spreadsheets—lives somewhere else. The link between the record and the data is often a file path that no longer works, or a reference only one person understands.
The metadata in your LIMS is valuable precisely because it can link to everything else. If you’ve never exported that metadata and matched it against your file shares, you’re missing the connection.
Instrument headers. Many instruments embed metadata in their output files. A mass spec file contains the acquisition parameters. An imaging file contains the stain type, magnification, and date. A plate reader export contains the plate layout.
This metadata is inside the files, not in a database. Often, nobody has extracted it. Nobody can search by it. But it’s the information you’d need to make the data useful.
File paths as metadata. This sounds strange, but it’s true: in many labs, the folder structure is the metadata. Projects/2024/CompoundX/Batch003/ tells you the project, the year, the compound, and the batch. That structure is inconsistent and undocumented, but it encodes real information.
Parsing the paths isn’t glamorous, but it works. We’ve done it for clients with petabytes of imaging data. The path structure, combined with file modification dates and instrument headers, is often enough to reconstruct the context.
How to find it in a week
You don’t need a big project to find your dark data. You need a metadata scan.
Day 1-2: List your data sources. Walk through the lab. Ask people where they store data. You want a list of every location: network drives, cloud buckets, instrument PCs, personal machines. Don’t judge or prioritize yet. Just list.
Day 3-4: Scan metadata. For each location, run a metadata scan. On a file share, this means walking the directory tree and collecting file names, paths, sizes, modification dates, and file types. You’re not reading the file contents—that would take weeks. You’re reading the file system.
For cloud storage (S3, GCS, Azure Blob), the metadata is even easier to extract. For databases and LIMS, export the record metadata without the blob data.
Day 5: Analyze. Load the metadata into a spreadsheet or a simple database. Look for:
- Total volume by location and file type
- Growth rate (compare modification dates)
- Files modified recently vs. untouched for years
- Naming patterns and folder structures
- Duplicates (same name, same size, different location)
You’ll often find things nobody knew about. A folder of imaging data from years ago. An instrument PC with files that were never copied. A naming convention that three people use differently.
Days 6-7: Report. Summarize what you found. Where is the data? How much is there? What’s growing? What’s at risk? What could be connected but isn’t?
This report is the starting point for any AI readiness work. You can’t connect what you can’t find.
The payoff
Dark data isn’t worthless data. It’s data that could be valuable if you could find it and connect it. The metadata scan tells you what you have. The next step is linking it together.
For a digital pathology company, we took a petabyte of slide images that nobody could search and made them searchable by stain type, tissue type, and customer. The metadata was already there—in the file headers and the folder paths. We just extracted it and linked it to their LIMS.
That’s the pattern. Find the data. Extract the metadata. Link it to records that give it meaning. Then you have something an AI assistant can actually use.
If this sounds like your lab, our AI Data Readiness assessment will map your dark data in detail and give you a plan for connecting it.
