dieterpy.scripts package

Submodules

dieterpy.scripts.gdx_handler module

This module contains functions that enable us to read GDX files and extract GAMS symbols, such as sets, variables, parameters and equations.

dieterpy.scripts.gdx_handler.gdx_get_set_coords(gams_dir: Optional[str] = None, filename: Optional[str] = None, setname: Optional[str] = None) List[str][source]

Based on the set name it provides a list with the set elements

Parameters
  • gams_dir (str, optional) – GAMS.exe path, if None the API looks at environment variables. Defaults to None.

  • filename (str, optional) – GDX filename. Defaults to None.

  • setname (str, optional) – name of the set. Defaults to None.

Raises

Exception – GDX file does not exist or is failed

Returns

a list with elements of a set found in the GDX file

Return type

list

dieterpy.scripts.gdx_handler.gdx_get_summary(gams_dir: Optional[str] = None, filename: Optional[str] = None) List[dict][source]

It returns a list of dictionaries. Each element of the list represents a symbol in the GDX file, every dictionary contains information of the symbols, such as, name, dimensions, type of symbol.

Parameters
  • gams_dir (str, optional) – GAMS.exe path, if None the API looks at environment variables. Defaults to None.

  • filename (str, optional) – GDX filename. Defaults to None.

Raises

Exception – GDX file does not exist or is failed.

Returns

a list of dictionaries with symbols’ info.

Return type

list[dict]

dieterpy.scripts.gdx_handler.gdx_get_symb_info(gams_dir: Optional[str] = None, filename: Optional[str] = None, symbol: Optional[str] = None)[source]

This function provides information of a symbol in a GDX file.

Parameters
  • gams_dir (str, optional) – GAMS.exe path, if None the API looks at Environment variables. Defaults to None.

  • filename (str, optional) – GDX filename. Defaults to None.

  • symbol (str, optional) – name of the symbol in the GDX file. Defaults to None.

Raises

Exception – GDX file does not exist or is failed

Returns

first position is a bool that indicates if symbol exists. Second position is a dictionary with symbol’s name, number of dimensions, dimension’s name in a list, type of symbol and description.

Return type

tuple

dieterpy.scripts.gdx_handler.gdx_get_symb_list(gams_dir: Optional[str] = None, filename: Optional[str] = None) List[str][source]

It returns a list of symbols’ names contained in the GDX file

Parameters
  • gams_dir (str, optional) – GAMS.exe path, if None the API looks at environment variables. Defaults to None.

  • filename (str, optional) – GDX filename. Defaults to None.

Raises

Exception – GDX file does not exist or is failed

Returns

a list of symbol’s names contained in the GDX file

Return type

list

dieterpy.scripts.gdx_handler.gdx_get_symb_recordnr(gams_dir: Optional[str] = None, filename: Optional[str] = None, symbolname: Optional[str] = None) tuple[source]

Return number of records in the requested symbol

Parameters
  • gams_dir (str, optional) – GAMS.exe path, if None the API looks at environment variables. Defaults to None.

  • filename (str, optional) – GDX filename. Defaults to None.

  • symbolname (str, optional) – name of the symbol. Defaults to None.

Raises

Exception – GDX file does not exist or is failed

Returns

It returns a tuple. The first position is bool if the symbol name exists in the GDX file. Second position an integer representing the number of records.

Return type

tuple

dieterpy.scripts.gdx_handler.gdx_get_symb_recordnr_from_list(gams_dir: Optional[str] = None, filenamelist: Optional[list] = None, symbolnamelist: Optional[list] = None) dict[source]

This function searches for each symbol of a list of symbols in the first GDX file of the list of files; if the current symbol is not present, it searches on the second file in the list. When a symbol is detected in a GDX file, it returns the symbol’s number of records. It returns a dictionary.

Parameters
  • gams_dir (str, optional) – GAMS.exe path, if None the API looks at environment variables. Defaults to None.

  • filenamelist (list[str], optional) – GDX files paths sorted in descending order of importance to find the symbols. Defaults to None.

  • symbolnamelist (list[str], optional) – list of symbols to search on GDX files. Defaults to None.

Raises
  • Exception – filenamelist can not be None

  • Exception – filenamelist must be a list

  • Exception – filenamelist is an empty list

  • Exception – A symbols is not present in any GDX file listed

Returns

Contains the number of records of each symbol in the GDX file that contains it.

Return type

dict

dieterpy.scripts.input_data module

This module contains the functions that help to process the input data

dieterpy.scripts.input_data.convert_par_var_dict(symbols_dict: Optional[dict] = None) tuple[source]

It convetrs the symbols to the format required to run GUSS tool.

Parameters

symbols_dict (dict, optional) – correspond to a iteration_main_dict[block][“par_var”]. Defaults to None.

Examples

dictionary of current symbols notation to guss dict nomenclature. e.g. example of a parameter and a variable: {‘ev_quant’: {‘body’: ‘ev_quant’, ‘dims’: (‘.’,)}, “N_TECH.lo(‘DE’,’pv’)”: {‘body’: ‘N_TECH.lo’, ‘dims’: (‘DE’, ‘pv’)}

Returns

2-element tuple containing

  • symb_guss_dict (dict): the dictionary contains symbols’ name as in the GAMS model. e.g. {0: {‘ev_quant’: {(‘.’,): 500.0},’N_TECH.lo’: {(‘DE’, ‘pv’): 5000.0}

  • symbols_guss_block_set_list (list): list of all symbols to be modified throughout all scenarios. e.g. [‘ev_quant’, ‘N_TECH’]

Return type

tuple

dieterpy.scripts.input_data.createModelCheckpoint(ws: gams.workspace.GamsWorkspace, opt: gams.options.GamsOptions, cp: gams.execution.GamsCheckpoint, model_dir_abspath: str, iter_data_dict: Optional[dict] = None, data_scen_key: str = 'NA') tuple[source]

Creates the initial checkpoint containing a model instance without solve command.

Parameters
  • ws (GamsWorkspace) – Base class of the gams namespace, used for initiating GAMS objects (e.g. GamsDatabase and GamsJob) by an “add” method of GamsWorkspace. Unless a GAMS system directory is specified during construction of GamsWorkspace, GamsWorkspace determines the location of the GAMS installation automatically. Aorking directory (the anchor into the file system) can be provided when constructing the GamsWorkspace instance. It is used for all file-based operations.

  • opt (GamsOptions) – Stores and manages GAMS options for a GamsJob and GamsModelInstance.

  • cp (GamsCheckpoint) – A GamsCheckpoint class captures the state of a GamsJob after the GamsJob.run method has been carried out. Another GamsJob can continue (or restart) from a GamsCheckpoint.

  • model_dir_abspath (str) – path of the folder where model.gms is hosted.

  • iter_data_dict (dict, optional) – dictioary that holds the time series iteration information. Defaults to None.

  • data_scen_key (str, optional) – identifier of time-series iteration data for a particular scenario. Defaults to “NA”.

Returns

2-element tuple containing

  • cp_file (str): GamsCheckpoint file path that contains defined model including the intended constraint w/o solve command.

  • main_gdx_file (str): contains the path of the resulting gdx file after the creation of the checkpoint.

Return type

tuple

dieterpy.scripts.input_data.defineGAMSOptions(opt: gams.options.GamsOptions, glob_feat_dc: dict, project_vars: dict, gdx_abspaths_dc: dict) gams.options.GamsOptions[source]

Defines global and control options and hands them over to the GAMS options object.

Parameters
  • opt (GamsOptions) – stores and manages GAMS options for a GamsJob and GamsModelInstance.

  • glob_feat_dc (dict) – contains GAMS model features and their activation case ‘*’ or ‘’.

  • project_vars (dict) – project variables collected from project_variables.csv.

  • gdx_abspaths_dc (dict) – contains all GDX input paths

Returns

updated GAMS options.

Return type

GamsOptions

dieterpy.scripts.input_data.gams_feat_node(gams_dir: Optional[str] = None, csv_path: Optional[str] = None, gdxoutputfolder: Optional[str] = None) str[source]

Creates a GDX file from CSV. The file converted is

Parameters
  • gams_dir (str, optional) – directory where gams.exe is located. Defaults to None.

  • csv_path (str, optional) – absolute path of the csv file. Defaults to None. Hint: The name of the file must fit the symbol name.

  • gdxoutputfolder (str, optional) – absolute path of the directory, without a “/” at the end. The name of the output file will be the same as the symbol with .gdx extension. Defaults to None.

Returns

path of the new GDX file

Return type

str

dieterpy.scripts.input_data.genIterationDict(project_vars: dict, path: str) tuple[source]

Function that creates the main iteration dict. First, a pandas DataFrame is imported from iteration_table.csv and then converted to a dictionary.

Parameters
  • project_vars (dict) – project variables collected from project_variables.csv.

  • path (str) – Input path to the folder where the main iteration csv files is hosted.

Raises

Exception – scenarios_iteration must be either “yes” or “no”. Check project_variables.csv

Returns

2-element tuple containing

  • iteration_main_dict (dict): Main iteration dictionary that hold all relevant information for the different scenario runs. The keys of the dictionary are the block numbers, and every block contains a dictionary with information for each contained scenario.

  • list_constraints (str): a list of constraints obtained from iteration_table.csv

Return type

tuple

dieterpy.scripts.input_data.genStringOptData(iter_data_dict: dict, key: str) str[source]

Generates a string that is compatible and used in the GAMS model to overwrite parameters for time series iteration.

Parameters
  • iter_data_dict (dict) – dictioary that holds the time series iteration information.

  • key (str) – identifier that selects the correct time series iteration “scenario”.

Returns

String to write in the GAMS opt object. It will enable GAMS model to select the intended time series from iter_data GAMS table.

Return type

str

dieterpy.scripts.input_data.generateInputGDX(input_path: str, config_folder: str, gdx_path: str, project_vars: dict) tuple[source]

Generates input gdx files if skip_import is ‘no’ in project_variables.csv.

Parameters
  • input_path (str) – path referring to folder location that contains input xlsx files.

  • config_folder (str) – path referring to folder location that contains project_variables.csv.

  • gdx_path (str) – path referring to folder location that contains input gdx files to be generated.

  • project_vars (dict) – project variables collected from project_variables.csv.

Returns

first position a list of active model features. Second position a dictionary with paths of new GDX input files.

Return type

tuple

dieterpy.scripts.input_data.getConfigVariables(config_folder: str) dict[source]

Imports configuration of the project variables and scenario variables from CSV file

Parameters

config_folder (str) – folder path of project_variables.csv location

Returns

Dictionary containing configuration variables

Return type

dict

dieterpy.scripts.input_data.getConstraintsdata(path: str) pandas.core.frame.DataFrame[source]

Get pandas dataframe of constraints_list.csv

Parameters

path (str) – path of the folder that contains constraints_list.csv.

Returns

pandas dataframe of constraints_list.csv

Return type

pd.DataFrame

dieterpy.scripts.input_data.getGDXoutputOptions(project_vars: dict) tuple[source]

Extract from project_variables.csv the formats on which the resulting GDX file will be converted. Options are CSV, PICKLE, and VAEX.

Parameters

project_vars (dict) – project variables collected from project_variables.csv.

Raises

Exception – features values must be “yes” or “no”

Returns

4-element tuple containing

  • csv_bool (bool): boolean

  • pickle_bool (bool): boolean

  • vaex_bool (bool): boolean

  • convert_cores (int): number of cores used to convert the symbols from GDX file to output formats.

Return type

tuple

dieterpy.scripts.input_data.getGlobalFeatures(config_folder: str, active_global_features: List[str]) dict[source]

This function prepares features that will be handed over to GAMS model as Global parameters. Creates dict with appropriate switch values for all global options. On-switch: ‘*’ Off-switch: “’’”

Parameters
  • config_folder (str) – Path referring to folder location that contains features_node_selection.csv.

  • active_global_features (List[str]) – It contains all features names that need to be activated in the GAMS model.

Returns

Contains switch information for global features.

Return type

dict

dieterpy.scripts.input_data.getGussVariables(project_vars: dict) tuple[source]

Collects from project_variables.csv the parameters related to the activation of GUSS tool.

Parameters

project_vars (dict) – project variables collected from project_variables.csv.

Raises
  • Exception – GUSS should be “yes” or “no” in project_variables.csv

  • Exception – GUSS_parallel should be “yes” or “no” in project_variables.csv

  • Exception – GUSS_parallel_threads must be an integer in project_variables.csv

Returns

3-element tuple containing

  • guss (bool): activation of GUSS tool.

  • guss_parallel (bool): run GUSS tool in parallel.

  • guss_parallel_threads (int): number CPUs used to run GUSS tool in parallel.

Return type

tuple

dieterpy.scripts.input_data.getIterableDataDict(input_path: str, output_path: str, project_vars: dict) tuple[source]

Generates a dictionary that holds the information for (data) time series iteration. e.g.: scenarios that alternate time series of solar capacity factors.

Parameters
  • input_path (str) – String that defines the input path for the excel file.

  • output_path (str) – String that defines the output path for the gdx file.

  • project_vars (dict) – project variables collected from project_variables.csv.

Returns

2-element tuple containing

  • iter_data_dict (str): Dictioary that holds the time series iteration information.

  • output_gdx_abspath (str): string absolute path of the output gdx file.

Return type

tuple

dieterpy.scripts.input_data.getTopographydata(path: str, project_vars: dict) pandas.core.frame.DataFrame[source]

Get pandas dataframe of ‘spatial’ sheet name of static_input.xlsx

Parameters
  • path (str) – Input path to the folder where the ‘static_input.xlsx’ is hosted.

  • project_vars (dict) – project variables collected from project_variables.csv.

Returns

pandas dataframe of ‘spatial’ sheet name of static_input.xlsx

Return type

pd.DataFrame

dieterpy.scripts.input_data.prepareGAMSAPI(working_directory: str) tuple[source]

Generates GAMS instances of workspace, checkpoint, and options.

Parameters

working_directory (str) – path of GAMS working directory. GAMS will host all working files for the optimization.

Returns

3-element tuple containing

  • ws (GAMS workspace object): Base class of the gams namespace, used for initiating GAMS objects (e.g. GamsDatabase and GamsJob) by an “add” method of GamsWorkspace. Unless a GAMS system directory is specified during construction of GamsWorkspace, GamsWorkspace determines the location of the GAMS installation automatically. Aorking directory (the anchor into the file system) can be provided when constructing the GamsWorkspace instance. It is used for all file-based operations.

  • cp (GAMS execution Checkpoint): A GamsCheckpoint class captures the state of a GamsJob after the GamsJob.run method has been carried out. Another GamsJob can continue (or restart) from a GamsCheckpoint.

  • opt (GAMS options object): Stores and manages GAMS options for a GamsJob and GamsModelInstance.

Return type

tuple

dieterpy.scripts.input_data.setCountryIteration(opt: gams.options.GamsOptions, block_iter_dc: dict, topography: pandas.core.frame.DataFrame) tuple[source]

It updates the activation of countries and transmission lines in GamsOptions.

Parameters
  • opt (GamsOptions) – updated GAMS options.

  • block_iter_dc (dict) – dictionary of each block with scenario parameters.

  • topography (pd.DataFrame) – spatial structure of the model. Defines which nodes are connected with which lines.

Returns

3-element tuple containing

  • opt (GamsOptions): updated GAMS options.

  • countries (str): countries to be written in the opt object.

  • lines (str): list of lines that will be used in that model run.

Return type

tuple

dieterpy.scripts.input_data.setDataIteration(opt: gams.options.GamsOptions, block_iter_dc: dict) tuple[source]

collect the key of time_series_scen in iteration_table.csv

Parameters
  • opt (GamsOptions) – updated GAMS options.

  • block_iter_dc (dict) – dictionary of each block with scenario parameters.

Returns

2-element tuple containing

  • opt (GamsOptions): updated GAMS options.

  • data_scen_key (str): identifier of time-series iteration data for a particular scenario.

Return type

tuple

dieterpy.scripts.input_data.writeConstraintOpt(opt: gams.options.GamsOptions, block_iter_dc: dict, list_constraints: List[str], itercon_dc: pandas.core.frame.DataFrame) dict[source]

Writes the selected constraints in the GAMS opt object for all scenarios that belong to a determined block.

Parameters
  • opt (GamsOptions) – updated GAMS options.

  • block_iter_dc (dict) – contains all symbols that will have different values through out the scenarios runs. A block contains a group of scenarios within same constraints or network topography.

  • list_constraints (List[str]) – List of constraints that exist in the block of scenarios.

  • itercon_dc (pd.DataFrame) – Panda that stores the different optional constraint see constraints_list.csv.

Raises

Exception – if the constraint obtained from block does not coincide with the ones in itercon_dc (constraints_list.csv).

Returns

dictionary whose keys are the types of constraints while values are a corresponding constraint name to hand over GAMS model. See constraints_list.csv to identify the type of constraints and contraint manes.

Return type

dict

dieterpy.scripts.input_data.writeCountryOpt(opt: gams.options.GamsOptions, countries: str, topos: pandas.core.frame.DataFrame) tuple[source]

Writes the sets ‘n’ (nodes) and ‘l’ (lines) in the GAMS options. Selects automatically the correct lines for the selected nodes.

Parameters
  • opt (GamsOptions) – updated GAMS options.

  • countries (str) – countries to be written in the opt object.

  • topos (pd.DataFrame) – spatial structure of the model. Defines which nodes are connected with which lines.

Returns

2-element tuple containing

  • countries (str): countries to be written in the opt object.

  • list_lines_export (str): list of lines that will be used in that model run.

Return type

tuple

dieterpy.scripts.output_data module

dieterpy.scripts.output_data.GDXpostprocessing(method='direct', input=None, cores_data=0, sysdir=None, csv_bool=True, pickle_bool=True, vaex_bool=True, base=None)[source]
dieterpy.scripts.output_data.gams_csvtovaex_parallel(L, queue, queue_lock, list_lock, scen_name, block, gams_dir, csv_bool, pickle_bool, vaex_bool, main_gdx, diffgdxfile=None)[source]

This function makes several actions to convert properly csv files that contain symbols. 1. when the gdx file is created from GUSS Tool, it contains only symbols that were modified from the original model. In this case, this fuction compares the original gdx (before GUSS tool scenario) with the resulting gdx. It opens symbol csv files and also read the original gdx. Then it compares the headers if they are identical or not. a. When they are identical, it converts the csv file to pandas dataframes and vaex (eventually). b. When the headers are different. It makes two steps, first compares the columns and second …

dieterpy.scripts.output_data.gams_gdxdiff(gams_dir=None, maingdx=None, scengdx=None, newfile=None, base=None)[source]
dieterpy.scripts.output_data.gams_gdxdumptocsv(gams_dir, scenario_dir_abspath, gdxfilename, symbname, base)[source]
dieterpy.scripts.output_data.gams_gdxdumptocsv_parallel(queue, queue_lock, gams_dir, csv_dir, gdxfilelist, base)[source]
dieterpy.scripts.output_data.get_solver_status(file)[source]
dieterpy.scripts.output_data.get_symb_from_features(feat_ingdx, symbfile)[source]
dieterpy.scripts.output_data.read_iteration_symbols(csvpath=None)[source]

Read csv file with the modified symbols for the iteration. csvpath: string absolute file path it returns a list with dictionaries, each dictionary contains the required data to execute one run. Example: {SymbolA: {dims element : value}, SymbolB: {dims element : value}}

dieterpy.scripts.output_data.solver_status_summary(method=None, input=None)[source]

dieterpy.scripts.report module

class dieterpy.scripts.report.CollectScenariosPerSymbol(paths=None, rng=None, cores=0)[source]

Bases: object

static add_scencols(scen, symbol, symbscen, shortscennames, loopitems, val_col, loopinclude)[source]
collectinfo(symbols=[])[source]
concatenation(symbol, flag, loopinclude, result_col, symblist)[source]
from_pkl_remove_df(path)[source]
static get_loopitems(data)[source]
static get_modifiers(scen, loopitems)[source]
join_all_symbols(result_col, loopinclude=False, warningshow=True)[source]
join_scens_by_symbol(symbol, result_col='v', loopinclude=False, warningshow=True)[source]

result_col: marginal or val self.data symbol

scen_load(path, symbol)[source]
static scenario_name_shortener(data)[source]
static showsymbols(data)[source]
symbol_dataframe_ready(scen, scenload_file_func, add_scencols_func, ts_func, shortscennames, loopitems, loopinclude, val_col, symbol)[source]
to_pickle(path, obj)[source]
static ts(df)[source]
class dieterpy.scripts.report.Symbol(name, value_type, unit, header_name, dims=None, symbol_type=None, index=['id'], symbol_handler=None)[source]

Bases: object

check_handler()[source]
check_index()[source]
check_inputs()[source]
check_value_type()[source]
concat(other)[source]
convert_dataframe(unit, mod=False)[source]
property df
property dfm
dfmdr(dim='h', nan=False, aggfunc='sum')[source]

dfmdr stands for dataframe with modifiers and dimension reduction dim are the list of dimension to be Reduced with sum(), default ‘h’ nan convert numeric columns, nan to -1 when value is True

dimreduc(dim, aggfunc='sum')[source]
fill_data()[source]
get(name)[source]
get_df()[source]
get_modifiers()[source]
reorganize(df, cols, original_name)[source]
set_df(df)[source]
show(unit=None, extend_header=None)[source]
showm(unit=None, extend_header=None)[source]
update_index(preferred_index)[source]
class dieterpy.scripts.report.SymbolsHandler(method, input=None)[source]

Bases: object

add_symbolfile(path)[source]
from_folder(folder_path=None)[source]
from_object(object)[source]
get_data(name, valuetype)[source]
get_loopitems()[source]
get_symbolnames()[source]
dieterpy.scripts.report.argmax(l)[source]
dieterpy.scripts.report.argmin(l)[source]
dieterpy.scripts.report.open_file(path)[source]
dieterpy.scripts.report.parallel_func(dc, queue=None, queue_lock=None, function=None, kargs={})[source]
dieterpy.scripts.report.parallelize(function=None, inputdict=None, nr_workers=1, **kargs)[source]

input is a dictionary that contains numbered keys and as value any object the queue contains tuples of keys and objects, the function must be consistent when getting data from queue

dieterpy.scripts.report.storagecycling(storage_in: dieterpy.scripts.report.Symbol, storage_out: dieterpy.scripts.report.Symbol) dieterpy.scripts.report.Symbol[source]

Symbol whose dataframe in the column ‘value’ has three possible options 0,1,2. Where 2 indicates storage cycling. This function gives a 1 if the flow occurs, otherwise, is 0 for each symbol (input and output flow). The both symbols are added, if at certain hour input and output flows have 1, the result will be 2.

Parameters
  • storage_in (Symbol) – Symbol storage input flow

  • storage_out (Symbol) – Symbol storage output flow

Returns

sto

dieterpy.scripts.runopt module

dieterpy.scripts.runopt.main()[source]

dieterpy.scripts.solve module

dieterpy.scripts.solve.clearModifiers(dcmodifier, maingdx)[source]
dieterpy.scripts.solve.guss_parallel(result, queue, queue_lock, print_lock, symbs, base, block)[source]
dieterpy.scripts.solve.guss_solve(queue, symbs, base, block)[source]
dieterpy.scripts.solve.scen_solve(scen_run, base, run, block)[source]

Function that imports all information, solves the DIETER model, and exports the results.

Parameters
  • workdir (string) – Working dictionary for the GAMS api.

  • cp_file (GAMS model instance) – Contains defined model including the intended constraint w/o solve command.

  • scen_run (dict) – Dictionary that holds information on parameters and variables for this specific block run.

  • str_country (string, optional) – String that holds the countries selected in the main iteration file. Used for the GDX-file name. The default is None.

  • str_lines (string, optional) – String that holds the lines selected according to the selected countries in the main iteration file. Used for the GDX-file name. The default is None.

  • str_constraints (string, optional) – String that holds the configs of the constraints selected in the main iteration file. Used for the GDX-file name. The default is None.

  • str_data (string, optional) – String that holds the scenario key for data iteration selected in the main iteration file. Used for the GDX-file name. The default is None.

Return type

None.

dieterpy.scripts.solve.setSymbolsValues(dcmodifier, dc, maingdx)[source]

dieterpy.scripts.util module

class dieterpy.scripts.util.OutputStream(tee=False, logfile=None)[source]

Bases: object

Output stream object for simultaneously writing to multiple streams.

Returns

[description]

Return type

[type]

flush()[source]

Needed for python3 compatibility.

write(message)[source]

Write messages to all streams.

Module contents

This module is the core of DIETERpy. It contains all functions to process the input data, the GAMS API to run and solve an optimization problem, and the functions that handle the resulting GDX files generated by GAMS.