Like I said, no real practical experience yet. The increase in complexity
that I fear is the loss of, e.g., writing arr[2,3] and having it not be the
element in the 2nd row and third column (i.e., the loss of a simple model
of how things are laid out). Maybe my fears are unfounded. Others don't
seem concerned it would seem.
I'll check out those packages that you mention.
Thanks,
Bob
On Sun, Oct 30, 2016 at 2:29 PM, Tim Holy <tim.h...@gmail.com> wrote:
> I'm afraid I still don't understand the claimed big increment in
> complexity. First, let's distinguish "generic offset arrays" from the
> OffsetArrays.jl package. If you're happy using OffsetArrays, you don't have
> to write your own offset-array type. Being able to use an established &
> tested package reduces your burden a lot, and you can ignore the second
> half of the devdocs page entirely.
>
> If you just want to *use* OffsetArrays.jl, the basic changes in coding
> style for writing indices-aware code are:
>
> - any place you used to call `size`, you probably want to call `indices`
> instead (and likely make minor adjustments elsewhere, since `incides`
> returns a tuple-of-ranges---but such changes tend to be very obvious);
> - check all uses of `similar`; some will stay as-is, other will migrate to
> `similar(f, inds)` style.
>
> In my experience, that's just about it. The devdocs goes into quite a lot
> of detail to explain the rationale, but really the actual changes are quite
> small. While you can't quite do it via `grep` and `sed`, to me that just
> doesn't seem complicated.
>
> Where the pain comes is that if you're converting old code, you sometimes
> have to think your way through it again---"hmm, what do I really mean by
> this index"? If your code had complicated indexing the first time you wrote
> it, unfortunately you're going to have to think about it carefully again;
> so in some cases, "porting" code is almost as bad as writing it the first
> time. However, if you write indices-aware code in the first place, in my
> experience the added burden is almost negligible, and in quite a few cases
> the ability to offset array indices makes things *easier* (e.g., "padding"
> an array on its edges is oh-so-much-clearer than it used to be, it's like a
> different world). That's the whole reason I implemented this facility in
> julia-0.5: to make life easier, not to make it harder. (Personally I think
> the whole argument over 0-based and 1-based indexing is stupid; it's the
> ability to use arbitrary indices that I find interesting & useful, and it
> makes most of my code prettier.)
>
> For examples of packages that use OffsetArrays, check the following:
> - CatIndices
> - FFTViews
> - ImageFiltering
>
> ImageFiltering is a mixed bag: there's a small performance penalty in a
> few cases (even if you use @unsafe) because that LLVM doesn't always
> optimize code as well as it could in principle (maybe @polly will help
> someday...). Because image filtering is an extraordinarily
> performance-sensitive operation, there are a few places where I had to make
> some yucky hand optimizations.
>
> Again, I'm very happy to continue this conversation---I'd really like to
> understand your concern, but without a concrete example I confess I'm lost.
>
> Best,
> --Tim
>
>
> On Sat, Oct 29, 2016 at 8:44 PM, Bob Portmann <bobportm...@gmail.com>
> wrote:
>
>> Thanks for the thoughtful response. I hope you'll tolerate one reply in
>> the "abstract".
>>
>> I am resisting the big change that occurred in 0.5. In 0.4 and earlier if
>> one declares an array as an `AbstractArray` in a function then one knew
>> that the indices were one based (a nice simple model, even if it is hated
>> by many). In 0.5, if one wants to write general code, then one has to
>> assume that arrays can have ANY indices. And one needs to write code using
>> more abstract tools. This seems to me to be a large cost in complexity for
>> the small subset of cases where offset indices are helpful. This is the
>> core issue to me.
>>
>> One way around this, it would seem, is to declare arrays as `::Array` and
>> not `::AbstractArray` in functions (if one want to be sure they are
>> `OneTo`). But then one gives up accepting many types of arrays that would
>> pose no problem (e.g, DistributedArrays with OneTo indices). Thus, I'm
>> proposing to have a high level abstract type that would capture all arrays
>> types that can be assumed to be `OneTo`. Then one can write library
>> functions against that type. This alone would help I think.
>>
>> The auto-conversion is an extra step that I thought might work since it
>> is (I think) low cost to convert an `OffsetArray` to a `OneTo` array . Thus
>> if you passed an OffsetArray to a function that takes the abstract OneTo
>> type (I kept its name AbstractArray above but that need not be its name)
>> you expect that if it returned a `similar` array it would be of `OneTo`
>> type. You would then have to convert it back to an OffsetArray type. It
>> would probably be easy to write one line functions to do this for you. If
>> it was sparse I assume it would remain so. It would be up to the writer of
>> the new OffsetArray type to declare what type it would convert to in the
>> OneTo array abstract tree.
>>
>> Does this help?
>>
>> Bob
>>
>> ps I don't have any concrete examples yet because OffsetArrays are so
>> scarce in the julia ecosystem. I'm just looking ahead to a world where it
>> will no longer be possible to assume AbstractArrays are OneTo. This seems
>> more painful than telling the 0-based crowd to code against 1-based arrays.
>>
>>
>> On Sat, Oct 29, 2016 at 10:40 AM, Tim Holy <tim.h...@gmail.com> wrote:
>>
>>> I should add, abstract discussions are all well and good, but I fully
>>> recognize you may have something very specific in mind and my answer
>>> simply
>>> fails to understand your concern. If you can give one or more concrete
>>> examples of where the current system is either bug-prone or a pain in the
>>> neck, I'd love to see what you mean.
>>>
>>> Best,
>>> --Tim
>>>
>>> On Thursday, October 27, 2016 5:21:26 PM CDT Bob Portmann wrote:
>>> > TL;DR I think the type system should take care of converting between
>>> > different types of arrays and thus free the user to code against the
>>> type
>>> > of array they want (OneTo, ZeroTo, Offset, etc).
>>> >
>>> > Maybe I have a deep mis-understanding of how the new generalized offset
>>> > arrays are suppose to work in Julia. If so I'd be happy to be set
>>> straight.
>>> > If not then I think the present system will lead to unnecessary
>>> complexity
>>> > and bug-ridden code. In the present system one can define array types
>>> that
>>> > are not one-based but can have arbitrary offsets. This is a great idea
>>> but
>>> > to make it work a very general system for indexing relative to the
>>> ends of
>>> > the array has been devised. If one writes a function that accepts an
>>> > `AbstractArray` then one MUST use this more general system or produce
>>> bugs
>>> > when e.g. a `ZeroToArray` is passed in. This puts a large burden on
>>> writers
>>> > of library code that works on arrays and I think is an unnecessary
>>> > complexity.
>>> >
>>> > Since Fortran arrays are a nice example let look at the situation in
>>> > Fortran. If I define an array in a function `real arr(-10:10)` then I
>>> would
>>> > index it using indices that range from -10 to 10. If I pass this into
>>> > another function that declared the array using its total size (usually
>>> > passed in separately or in a parameter statement) `real arr(21)` then
>>> in
>>> > that subroutine one would index the array using "normal" indices that
>>> range
>>> > from 1 to 21. I.E., you state in the function how you want to treat the
>>> > array and are not forced to work with in offset form unless you want
>>> to. In
>>> > my opinion, this is what makes using offset arrays in Fortran sane and
>>> is
>>> > the key thing that Julia should try to emulate.
>>> >
>>> > One way to get this behavior in Julia would be to use the type system
>>> and
>>> > `convert`. Since `AbstractArray` has meant until 0.5 a `OneTo` array I
>>> > think it is wise that it remain that way so that all old code will work
>>> > unchanged (even with the new array types). Thus, I propose adding a new
>>> > top-level type above AbstractArray type to capture all arrays something
>>> > like:
>>> >
>>> > ```
>>> > Array{T,N} <: DenseArray{T,N} <: AbstractArray{T,N} <:
>>> > AbstractAllArray{T,N} <: Any
>>> > ```
>>> >
>>> > And the offset arrays would be subtypes of `AbstractAllArrays` on a
>>> > different branch
>>> >
>>> > ```
>>> > OffsetArray{T,N} <: AbstractOffsetArray{T,N} <: AbstractAllArray{T,N}
>>> <: Any
>>> > ```
>>> >
>>> > And similarly for `ZeroToArray`.
>>> >
>>> > Then, if one declares an Array as
>>> >
>>> > ```
>>> > function func(arr::AbstractArray)
>>> > ```
>>> >
>>> > one can safely assume it is a 1-based Array inside `func`. If an offset
>>> > array is passed into `func` then it is automatically converted to a
>>> `OneTo`
>>> > array inside `func`. This is the key point of this proposal and is
>>> similar
>>> > to the auto-conversion of, e.g., Int to Floats in a function call.
>>> > Similarly if one declares an array as a `ZeroToArray` in a function and
>>> > passes in an `Array` then it will be converted to a `ZeroToArray` in
>>> the
>>> > function. Some conversions would need more information and thus would
>>> be
>>> > disallowed (e.g., `Array` to `OffsetArray`). I think this system would
>>> go a
>>> > long way towards making `ZeroToArray`s and `OffsetArray`s simple to
>>> use in
>>> > Julia w/o using the generalized indexing techniques introduced in Julia
>>> > 0.5. Note that it would still be possible to use the `AbstractAllArray`
>>> > type and accept all arrays w/o conversion and use the generalized
>>> indexing
>>> > techniques.
>>> >
>>> > I'm curious what people think about this.
>>> >
>>> > Bob
>>> >
>>> > ps Paste into github to see in formatted form. I'm not sure how to get
>>> that
>>> > in an email.
>>>
>>>
>>>
>>
>
