It is part of a larger project aiming at processing data according to a given algorithm Do you have any comments or any enhancing recommendations on the code?
Thanks. Le dim. 21 janv. 2024 à 18:28, Thomas Passin via Python-list < python-list@python.org> a écrit : > On 1/21/2024 11:54 AM, marc nicole wrote: > > Thanks for the reply, > > > > I think using a Pandas (or a Numpy) approach would optimize the > > execution of the program. > > > > Target cells could be up to 10% the size of the dataset, a good example > > to start with would have from 10 to 100 values. > > Thanks for the reformatted code. It's much easier to read and think about. > > For say 100 points, it doesn't seem that "optimization" would be much of > an issue. On my laptop machine and Python 3.12, your example takes > around 5 seconds to run and print(). OTOH if you think you will go to > much larger datasets, certainly execution time could become a factor. > > I would think that NumPy arrays and/or matrices would have good potential. > > Is this some kind of a cellular automaton, or an image filtering process? > > > Let me know your thoughts, here's a reproducible example which I > formatted: > > > > > > > > from numpy import random > > import pandas as pd > > import numpy as np > > import operator > > import math > > from collections import deque > > from queue import * > > from queue import Queue > > from itertools import product > > > > > > def select_target_values(dataframe, number_of_target_values): > > target_cells = [] > > for _ in range(number_of_target_values): > > row_x = random.randint(0, len(dataframe.columns) - 1) > > col_y = random.randint(0, len(dataframe) - 1) > > target_cells.append((row_x, col_y)) > > return target_cells > > > > > > def select_contours(target_cells): > > contour_coordinates = [(0, 1), (1, 0), (0, -1), (-1, 0)] > > contour_cells = [] > > for target_cell in target_cells: > > # random contour count for each cell > > contour_cells_count = random.randint(1, 4) > > try: > > contour_cells.append( > > [ > > tuple( > > map( > > lambda i, j: i + j, > > (target_cell[0], target_cell[1]), > > contour_coordinates[iteration_], > > ) > > ) > > for iteration_ in range(contour_cells_count) > > ] > > ) > > except IndexError: > > continue > > return contour_cells > > > > > > def create_zipf_distribution(): > > zipf_dist = random.zipf(2, size=(50, 5)).reshape((50, 5)) > > > > zipf_distribution_dataset = pd.DataFrame(zipf_dist).round(3) > > > > return zipf_distribution_dataset > > > > > > def apply_contours(target_cells, contour_cells): > > target_cells_with_contour = [] > > # create one single list of cells > > for idx, target_cell in enumerate(target_cells): > > target_cell_with_contour = [target_cell] > > target_cell_with_contour.extend(contour_cells[idx]) > > target_cells_with_contour.append(target_cell_with_contour) > > return target_cells_with_contour > > > > > > def create_possible_datasets(dataframe, target_cells_with_contour): > > all_datasets_final = [] > > dataframe_original = dataframe.copy() > > > > list_tuples_idx_cells_all_datasets = list( > > filter( > > lambda x: x, > > [list(tuples) for tuples in > > list(product(*target_cells_with_contour))], > > ) > > ) > > target_original_cells_coordinates = list( > > map( > > lambda x: x[0], > > [ > > target_and_contour_cell > > for target_and_contour_cell in target_cells_with_contour > > ], > > ) > > ) > > for dataset_index_values in list_tuples_idx_cells_all_datasets: > > all_datasets = [] > > for idx_cell in range(len(dataset_index_values)): > > dataframe_cpy = dataframe.copy() > > dataframe_cpy.iat[ > > target_original_cells_coordinates[idx_cell][1], > > target_original_cells_coordinates[idx_cell][0], > > ] = dataframe_original.iloc[ > > dataset_index_values[idx_cell][1], > > dataset_index_values[idx_cell][0] > > ] > > all_datasets.append(dataframe_cpy) > > all_datasets_final.append(all_datasets) > > return all_datasets_final > > > > > > def main(): > > zipf_dataset = create_zipf_distribution() > > > > target_cells = select_target_values(zipf_dataset, 5) > > print(target_cells) > > contour_cells = select_contours(target_cells) > > print(contour_cells) > > target_cells_with_contour = apply_contours(target_cells, > contour_cells) > > datasets = create_possible_datasets(zipf_dataset, > > target_cells_with_contour) > > print(datasets) > > > > > > main() > > > > Le dim. 21 janv. 2024 à 16:33, Thomas Passin via Python-list > > <python-list@python.org <mailto:python-list@python.org>> a écrit : > > > > On 1/21/2024 7:37 AM, marc nicole via Python-list wrote: > > > Hello, > > > > > > I have an initial dataframe with a random list of target cells > > (each cell > > > being identified with a couple (x,y)). > > > I want to yield four different dataframes each containing the > > value of one > > > of the contour (surrounding) cells of each specified target cell. > > > > > > the surrounding cells to consider for a specific target cell are > > : (x-1,y), > > > (x,y-1),(x+1,y);(x,y+1), specifically I randomly choose 1 to 4 > > cells from > > > these and consider for replacement to the target cell. > > > > > > I want to do that through a pandas-specific approach without > > having to > > > define the contour cells separately and then apply the changes on > the > > > dataframe > > > > 1. Why do you want a Pandas-specific approach? Many people would > > rather > > keep code independent of special libraries if possible; > > > > 2. How big can these collections of target cells be, roughly > speaking? > > The size could make a big difference in picking a design; > > > > 3. You really should work on formatting code for this list. Your > code > > below is very complex and would take a lot of work to reformat to the > > point where it is readable, especially with the nearly impenetrable > > arguments in some places. Probably all that is needed is to replace > > all > > tabs by (say) three spaces, and to make sure you intentionally break > > lines well before they might get word-wrapped. Here is one example I > > have reformatted (I hope I got this right): > > > > list_tuples_idx_cells_all_datasets = list(filter( > > lambda x: utils_tuple_list_not_contain_nan(x), > > [list(tuples) for tuples in list( > > itertools.product(*target_cells_with_contour)) > > ])) > > > > 4. As an aside, it doesn't look like you need to convert all those > > sequences and iterators to lists all over the place; > > > > > > > (but rather using an all in one approach): > > > for now I have written this example which I think is not Pandas > > specific: > > [snip] > > > > -- > > https://mail.python.org/mailman/listinfo/python-list > > <https://mail.python.org/mailman/listinfo/python-list> > > > > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
