> > > here's a snippet. if it attack you put your hand in front of it and > > > say "down!". it's just the intro it's not even the area i'm working on > > > > # ==> i think work ended up in the read_many function despite > > the read_many_tensors function existing > > # ==> basically, one outputs partial tensors unless there is a > > region of all-cached or all-uncached > > # ==> the special cases are for runs of 0s on the left and > > right and can be ignored to start > > updated this line to: > # ==> the special cases are for disk exhaustion and runs of 0s > on the left and right, and can be ignored to start > > > > > def read_many(self, offset_lengths, progress, validate_sorted=True): > > if validate_sorted: > > sorted_offset_lengths = list(offset_lengths) > > sorted_offset_lengths.sort() > > assert sorted_offset_lengths == offset_lengths > > OP_FETCH, OP_PLACE, OP_OUTPUT = 1, 2, 4 > > offset_length_tail_idx_ops = > > torch.zeros([offset_lengths.shape[0]*2, 5]) > > OFFSET, LENGTH, TAIL, IDX, OP = > > range(offset_length_tail_ops.shape[-1]) > > op_ct = 0 > > tails = (offset_lengths[:,0] + > > offset_lengths[:,1]).clamp(max=len(self.mmap)) > > aligned_offsets > > = offset_lengths[:,0] // self.blksize; aligned_offsets *= self.blksize > > aligned_tails = (tails - 1); aligned_tails //= > > self.blksize; aligned_tails += 1; aligned_tails *= self.blksize; > > torch.clamp(aligned_tails, max=self.size(), out=aligned_tails) > > cls = type(self.fetchers) > > avail_disk_space = (psutil.disk_usage(self.fn).free + > > cls.sparse_usage) * self.fetchers.usage_frac - cls.sparse_usage > > min_hole = 0 > > pbar = range(len(offset_lengths)) > > if progress: > > pbar = tqdm.tqdm(pbar, total=len(offset_lengths), > > desc=progress, leave=False, unit='rd') > > idx = 0 > > while idx < len(offset_lengths):
i'm guessing that basically this code is mostly implemented at this point and mostly needs to be cleaned up, fuzzed, and extensively further cleaned up and fixed, to address how it was made maybe i should assume the ops are placed, write the lines that use them, and then fuzz it, and then i can continue by rote. assuming all the logical problems have somehow been solved and i can reconstruct the solutions when encountering mistakes
