Managing survey data...what am I doing wrong?

When I started data collecting (many moons ago) we used the standard format with an ASCII file and a RAW data file.  I use worksheets for section breakdowns and block breakdowns in subdivisions so there is no double calculations etc.  These ASCII files are getting quite large nowadays with some approaching over 20,000 points (data dating back to 1990).  These ASCII's contain all the points from control points, to property corner monuments, to field located items (edge of pavement, power poles, house corners etc.) to calculations (property corner calcs, deed calcs etc.).  My question is, how do many of you manage this data?  Do any of you remove the points not needed in future work such as improvements and location points?  Am I just being "picky" here?  I'd like to think I've got several years left in me surveying, but how much bigger are these files going to get?  I know storage capacity of data collectors has increased, but a couple of our files are so big you can only have 1 of them in the data collector at a time.  Just trying to get ideas on what others do with their data.

Thanks,
V

ASCII txt files are very small as survey files go. But it  looks on the surface like you are simply using the wrong tool to manage your data. File management is the domain of the PC at the least and a server at the best. Nearly everyone I know manages jobs by name and date. Basic file mangement in Windows Explorer is fairly easy. You just need a naming convention and off you go. Storage is pretty cheap these days. You can get an external hard drive with a terabyte capacity for a little more than a hundred dollars.

If your data collector is challenged by filesize, you can get a laptop for around $350 these days. Size the files as you need them. You can carry a sizeable database around on a laptop and upload as needed to a data collector.

Retention schedules are whatever you think they ought to be. Only you can decide how long your data maintains its value. Just about any work you have done is probably worth more than a hundred dollar external back up drive. A lot more if you ever have to go to court and cannot produce it.

The hard, that is time consuming part, is setting up your storage schema and organizing the data. It sounds as though you need to get started on that.
MLB

What do you think about eliminating all located points in ASCII files?  What I mean is points that are not survey control, calculations or monuments?

One of the reasons I don't is because liability lasts longer than the Energizer Bunny. People do things to your work after you complete your part. Sometimes they make inapprpriate edits which can cause resultant products to be flawed. When this happens, the Surveyor often makes it to the top of the list of "usual suspects". Keeping the original raw data file is Exhibit A in your defense.
MLB

I agree with MLB's comment that "it looks on the surface like you are simply using the wrong tool to manage your data. File management is the domain of the PC at the least and a server at the best. Nearly everyone I know manages jobs by name and date." and woder why you'd need all those points in your data collector (DC).

Do you actually manage your data in the office, as per MLB?
Or does your office survey/CAD software not handle the "large" files adequately? If that's the case, you need better survey/CAD software. Even free software like Copan can easily handle much larger files than your 6600 point example.

Eliminate points? No! Keep all your data in your office, somehow, and, when needed, extract copies of just the control points (or whatever you really need) from the office system, to place on the DC, for each trip back to the field.

your job data base is set up such that for a particular large all inclusive area(s)local government area that you add all data?

For what it is worth I have township/suburbs directories that I then add a job in a sub directory also including street name, lot number directory to then be able to add all files associated with that job.

I know that we have far more coordinated reference material that is nearby our surveys, so that makes a difference in that we can localize the job.

However in your case you can still have separate jobs, but create a master control directory build up of specific major control points, either in one file or a series of files in keeping with survey regions. So when you wish to carry out a survey in an area you import relevant control points from the master file to the particular job file. Then on completion of job you then export back to master file any new control points created from the control data base.

I guess the degree of difficulty to establish a master control data base is if you have uniquely numbered control points in a block of reserved point numbers or with a unique point code.
 
 
This way you only down load small files into data recorder that include control and point set out data if already orientated the set out data.

I suspect many surveyors have never given serious thought to job management until they return years later to a nearby site to survey.
 
Of course you need to back up files.


RADU